Fun with text files
-
I have a 2GB text file with about 2.3 million records It is updated weekly.
The data is in a 200 character wide fixed width format.
In order to do what needs done, it is being brought into SQL with a
BULK INSERT
.
That step takes just under a minute on my machine. Not aware of any better method for this step.
The next thing it does is execute a stored proc that inserts each line into a new table with everything substringed out into pieces. that only taker another 40 seconds or so.
Then do a ton of recursive querying to link things together under certain criteria. Only 2 minutes per day there.
Related: looking at code your wrote 6 years ago is sometimes scary.
-
@JaredBusch said in Fun with text files:
Related: looking at code your wrote 6 years ago is sometimes scary.
Especially when it works and you're not exactly sure why! lol.
-
at "200 character width" I started thinking about the COBOL class I took in college.