From jrwright at cs.ubc.ca Fri Jul 24 01:30:02 2009 From: jrwright at cs.ubc.ca (James Wright) Date: Thu, 23 Jul 2009 18:30:02 -0700 Subject: [cl-sqlite-devel] [patch] timeout support Message-ID: <5bd1017b0907231830k7a691471h7acf008b2c1279c5@mail.gmail.com> Hi, I've been using cl-sqlite to dump results from parallel processes into a sqlite db. The problem with this is that sometimes two processes will attempt to write to the file at the same time. The sqlite library will retry, but it will only do so if a timeout has been set on the connection. I've added timeout/retry support to cl-sqlite, and I thought it might be useful to others, so I've attached a patch. I don't have a platform-independent unit test, since I believe it needs threads to test properly, but I've attached (in 'sqlite-noodling.lisp') a file that reproduces the problem (function `tst1') and shows that the fix actually fixes it (`tst2'). James -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Added-support-for-timed-wait-for-locked-databases-a.patch Type: text/x-diff Size: 3388 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sqlite-noodling.lisp Type: application/octet-stream Size: 1385 bytes Desc: not available URL: From kalyanov.dmitry at gmail.com Mon Jul 27 05:48:58 2009 From: kalyanov.dmitry at gmail.com (Kalyanov Dmitry) Date: Mon, 27 Jul 2009 09:48:58 +0400 Subject: [cl-sqlite-devel] [patch] timeout support Message-ID: <200907270949.02610.Kalyanov.Dmitry@gmail.com> On Friday 24 July 2009 05:30:02 James Wright wrote: > Hi, > > I've been using cl-sqlite to dump results from parallel processes into > a sqlite db. The problem with this is that sometimes two processes > will attempt to write to the file at the same time. The sqlite > library will retry, but it will only do so if a timeout has been set > on the connection. I've added timeout/retry support to cl-sqlite, and > I thought it might be useful to others, so I've attached a patch. > > I don't have a platform-independent unit test, since I believe it > needs threads to test properly, but I've attached (in > 'sqlite-noodling.lisp') a file that reproduces the problem (function > `tst1') and shows that the fix actually fixes it (`tst2'). > > James Thanks! I've applied the patch to git. I've changed test to use bordeaux-threads (threading compatibility library). Did you try tests on a multi-core machine? On my single-core machine tests start failing only with larger amount of writers and threads. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From kalyanov.dmitry at gmail.com Mon Jul 27 05:43:14 2009 From: kalyanov.dmitry at gmail.com (Kalyanov Dmitry) Date: Mon, 27 Jul 2009 09:43:14 +0400 Subject: [cl-sqlite-devel] [patch] timeout support In-Reply-To: <5bd1017b0907231830k7a691471h7acf008b2c1279c5@mail.gmail.com> References: <5bd1017b0907231830k7a691471h7acf008b2c1279c5@mail.gmail.com> Message-ID: <200907270943.18602.Kalyanov.Dmitry@gmail.com> On Friday 24 July 2009 05:30:02 James Wright wrote: > Hi, > > I've been using cl-sqlite to dump results from parallel processes into > a sqlite db. The problem with this is that sometimes two processes > will attempt to write to the file at the same time. The sqlite > library will retry, but it will only do so if a timeout has been set > on the connection. I've added timeout/retry support to cl-sqlite, and > I thought it might be useful to others, so I've attached a patch. > > I don't have a platform-independent unit test, since I believe it > needs threads to test properly, but I've attached (in > 'sqlite-noodling.lisp') a file that reproduces the problem (function > `tst1') and shows that the fix actually fixes it (`tst2'). > > James Thanks! I've applied the patch to git. I've changed test to use bordeaux-threads (threading compatibility library). Did you try tests on a multi-core machine? On my single-core machine tests start failing only with larger amount of writers and threads. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: From james at chumsley.org Mon Jul 27 14:43:49 2009 From: james at chumsley.org (James Wright) Date: Mon, 27 Jul 2009 07:43:49 -0700 Subject: [cl-sqlite-devel] [patch] timeout support In-Reply-To: <200907270949.02610.Kalyanov.Dmitry@gmail.com> References: <200907270949.02610.Kalyanov.Dmitry@gmail.com> Message-ID: <5bd1017b0907270743g44f014acnd25217bf88567421@mail.gmail.com> > Thanks! You're very welcome. And thank-you for a very useful library, as I should have said at the outset. :) > Did you try tests on a multi-core machine? On my single-core machine tests > start failing only with larger amount of writers and threads. Yes, I did indeed run my tests on a dual core machine. I was surprised that it took such a small number of threads to fail, but it never occurred to me that multiple cores were the reason. James