February 16, 2019 at 2:22 am
#16949
Keymaster
In most cases relational databases do not handle batch processing of large amounts of data very well, not the least of which is RDBMS logging requirements, which set a maximum limit on data changes in one transaction.
Moving lots of data in is an operational concern, so an operational approach is a good and logical choice.
Break your file up,externally to the DB (using command line tools), into many smaller files of say 10K each and load them as you would the single large file – keep the chunking logic outside the DB