| *** PaulM05 has joined #ingres | 01:09 | |
| *** rossand has joined #ingres | 01:26 | |
| *** ChanServ sets mode: +o rossand | 01:26 | |
| * grantc is away: Away | 02:31 | |
| *** atrofast has joined #ingres | 03:54 | |
| *** KermitTheFragger has joined #ingres | 03:57 | |
| *** dyki has joined #ingres | 04:11 | |
| *** PaulM05 is now known as PaulM05_lunch | 04:29 | |
| *** PaulM05_lunch has quit IRC | 04:50 | |
| *** cthibert has joined #ingres | 05:26 | |
| *** awatkins_ has joined #ingres | 05:38 | |
| awatkins_ | hello, i understand that for copy, the best performance is using the byte copy, but for some of our tables it creates an unnecessarily large file becaus eof hte amount of nulls we have. however, loading up a csv file is significantly slower, does anyone know of a happy medium? | 05:41 |
|---|---|---|
| *** KermitTheFragger has quit IRC | 05:49 | |
| *** zxiiro has quit IRC | 05:51 | |
| *** dyki has quit IRC | 05:58 | |
| *** gerhard has joined #ingres | 06:09 | |
| *** mull has joined #ingres | 06:26 | |
| *** zxiiro has joined #ingres | 06:28 | |
| *** ChanServ sets mode: +v zxiiro | 06:28 | |
| *** PaulM05 has joined #ingres | 06:35 | |
| DarylM | member:awatkins_: Nulls in a binary copy should not create an unusually large output file. Unless you are talking about a "copy table" in ascii mode. | 06:40 |
| *** rossand has quit IRC | 06:43 | |
| awatkins_ | DarylM: the command we're using is copy <table> () into 'file'; the output is binary, however, for a varchar(255) field, the output padded for hte full 255 characters with \000, which makes the file quite large, since we have a lot of columns | 06:54 |
| pboro | awatkins_, you can overcome that problem by using ascii copy | 06:54 |
| awatkins_ | pboro: right, but then isn't the reimport a fair amount slower? | 06:55 |
| pboro | awatkins_, I suggest you try it out at least, for us it wasnt | 06:56 |
| DarylM | It isn't the nulls, it is a strict image copy of the data. There is a trick. You can use Unix named pipes as the "copy table" output file and route the binary data straight to gzip | 06:56 |
| pboro | awatkins_, also remember, than those varchar(255)s take the full 255 for every row in your db if you are not using compression | 06:56 |
| awatkins_ | we're actually using vectorwise, and i'm pretty sure it does heavy compression | 06:57 |
| DarylM | Process 1: "copy table ..... into '/tmp/namedpipe' Process 2: gzip -c </tmp/namedpipe >yourfile | 06:57 |
| pboro | awatkins_, ah, not familiar with vectorwise | 06:58 |
| awatkins_ | DarylM: huh, actually thats pretty cool, and then to reimport just pipe the output of gzip back to 'sql'? | 06:58 |
| DarylM | right. Just reverse the process | 06:58 |
| DarylM | It is a bit tricky and you incur the CPU overhead, but it works quite well | 06:59 |
| DarylM | The only other common option is "-1" on the gzip. | 06:59 |
| DarylM | It reduces CPU load substantially at minimum increases to disk space use. | 06:59 |
| DarylM | Caveat: I have not tested that on a Vectorwise install but It should work fine. | 07:00 |
| *** Alex| has quit IRC | 07:01 | |
| *** Alex| has joined #ingres | 07:01 | |
| *** ChanServ sets mode: +o Alex| | 07:01 | |
| *** DarylM has left #ingres | 07:01 | |
| *** DarylM has joined #ingres | 07:02 | |
| * DarylM hates it when he closes the wrong window | 07:02 | |
| awatkins_ | DarylM: after talking with one of hte engineers, the copy in/out was ripped directory from 10.0, so it should work | 07:08 |
| *** Alex| has quit IRC | 07:09 | |
| *** KermitTheFragger has joined #ingres | 07:20 | |
| *** gerhard has quit IRC | 08:02 | |
| awatkins_ | has anyone every exported using hte binary copy, and then on reimport getting Unexpected End of File on row 1? | 08:41 |
| *** Mud has joined #ingres | 08:53 | |
| *** grantc has quit IRC | 09:33 | |
| *** KermitTheFragger has quit IRC | 10:04 | |
| *** PaulM05 has quit IRC | 10:11 | |
| *** Alex| has joined #ingres | 10:23 | |
| *** ChanServ sets mode: +o Alex| | 10:23 | |
| *** Mud has quit IRC | 11:22 | |
| *** Mud has joined #ingres | 11:28 | |
| *** cthibert has left #ingres | 13:15 | |
| *** Alex| has quit IRC | 13:16 | |
| *** Mud has quit IRC | 13:28 | |
| *** rossand has joined #ingres | 13:29 | |
| *** ChanServ sets mode: +o rossand | 13:29 | |
| *** zxiiro has quit IRC | 13:32 | |
| *** zxiiro has joined #ingres | 14:21 | |
| *** ChanServ sets mode: +v zxiiro | 14:21 | |
| *** atrofast has quit IRC | 15:26 | |
| *** mull has quit IRC | 15:39 | |
| *** mull has joined #ingres | 16:27 | |
| *** rossand has quit IRC | 17:17 | |
| *** awatkins_ has quit IRC | 17:45 | |
| *** mull has quit IRC | 19:10 | |
| *** cytrinox has quit IRC | 19:24 | |
| *** cytrinox has joined #ingres | 19:25 | |
| *** Alex| has joined #ingres | 21:35 | |
| *** ChanServ sets mode: +o Alex| | 21:35 | |
| *** Mud has joined #ingres | 21:36 | |
| *** rossand has joined #ingres | 22:07 | |
| *** ChanServ sets mode: +o rossand | 22:07 | |
| *** Alex| has quit IRC | 22:08 | |
| *** Alex| has joined #ingres | 22:10 | |
| *** ChanServ sets mode: +o Alex| | 22:10 | |
| *** Alex| has quit IRC | 22:14 | |
| *** bonro011 has joined #ingres | 22:59 | |
| *** bonro01 has quit IRC | 23:01 | |
| *** bonro01 has joined #ingres | 23:03 | |
| *** bonro011 has quit IRC | 23:05 | |
| *** Mud has quit IRC | 23:22 | |
| *** Mud has joined #ingres | 23:28 | |
| *** rossand has quit IRC | 23:41 | |
| *** grantc has joined #ingres | 23:53 | |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!