Making Betfair applications in JavaScript is now possible

[AlgoTrader]

I am not kidding. There is no need in browser and it is no way related to browser. There is node.js framework that makes it possible. Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices.

There is betfair-sports-api for Node.js. Here is the demo:

C:\Node>npm install betfair-sports-api
npm http GET https://registry.npmjs.org/betfair-sports-api
npm http 200 https://registry.npmjs.org/betfair-sports-api
npm http GET https://registry.npmjs.org/betfair-s...ir-sports-api-
0.1.1.tgz
npm http 200 https://registry.npmjs.org/betfair-s...ir-sports-api-
0.1.1.tgz
npm http GET https://registry.npmjs.org/dom-js
npm http GET https://registry.npmjs.org/async
npm http 304 https://registry.npmjs.org/async
npm http 304 https://registry.npmjs.org/dom-js
npm http GET https://registry.npmjs.org/sax
npm http 304 https://registry.npmjs.org/sax
betfair-sports-api@0.1.1 ./node_modules/betfair-sports-api
├── async@0.1.18
└── dom-js@0.0.6

C:\Node>node node_modules\betfair-sports-api\test\get_all_markets.js
===== Logging in to Betfair... =====
Session token is: [skipped]
Login error INVALID_USERNAME_OR_PASSWORD

C:\Node>SET BF_LOGIN=user

C:\Node>SET BF_PASSWORD=password

C:\Node>node node_modules\betfair-sports-api\test\get_all_markets.js
===== Logging in to Betfair... =====
Session token is: [skipped]
Logged in OK
===== Get available tennis matches =====
there are 134 markets
action: getAllMarkets error: null duration: 0.435
Family Circle Cup 2012\Third Round Matches\Voskoboeva v Stosur
Family Circle Cup 2012\Third Round Matches\Erakovic v S Williams
Family Circle Cup 2012\Third Round Matches\Bartoli v Hercog
Family Circle Cup 2012\Third Round Matches\Petrova v Pavlyuchenkova
Family Circle Cup 2012\Third Round Matches\Zvonareva v Voegele
Family Circle Cup 2012\Third Round Matches\Lisicki v Shvedova
Family Circle Cup 2012\Third Round Matches\Wozniak v Safarova
Family Circle Cup 2012\Third Round Matches\V Williams v An Rodionova
===== Logging out... =====
Logged out OK

C:\Node>

The source code is there https://github.com/AlgoTrader/betfai...all_markets.js

What are the reasons to use JavaScript?
- Development is easy and rapid, it is faster than C++, C# or Java
- It is really fast, much faster than most of the other scipting languages
- Fully asuncronous, placing inplay bet does not stops other markets to be refreshed, thanks to asyncronous Node.js
- Cross-platform: develop on Windows, deploy on Linux, present on MacOSX
- Hosting friendly, place the apps on VPS in UK and get fast ping

I switched to JavaScript from C++ as I felt I spend too much time fighting with the language: memory management, much verbose code, figting with callbacks (signals/slots in QT) and too static typing. Node.js is based on Google V8 engine that compiles JS into native code (that is why it so fast) but I still have garbage collector, dynamic types, literal objects and many other benefits of JavaScript.

I was an opponent of JavaScript some time ago, but I had to change my mind. While JavaScript can be really ugly when embedded in HTML pages it can be beatiful in Node.js environment.

The betfair-sports-api for Node was developed in two-three weeks and I got lots of fun programming it. There are 10 "tests" in the package but I really meant rather "examples". Hopefully my work can be useful for many as it is free, open-source and licenced under permissive licence.

PS. Performance testing on a commpdity PC using betfair-sports-api 0.1.4 showed 260 keepAlives a second (I sent 1000 keepAlives and got 1000 replies) in 3.8 seconds. I tested in on my Windows box, linux results can be better

[elmariachi]

Very interesting indeed. Are you using the js library in production code? Or are you currently testing only? I'm asking since you state on the github page that: The betfair-sports-api is pretty usable not but not tested in production.

[AlgoTrader]

Originally Posted by elmariachi

Very interesting indeed. Are you using the js library in production code? Or are you currently testing only? I'm asking since you state on the github page that: The betfair-sports-api is pretty usable not but not tested in production.

The code works and works great. The first production app is being written. The first results are exellent, but there are several minor problems. All of them are certainly not the serouos problems.

The most important problem is to ensure the pool of persistent connections. Have a look:

===== Logging in to Betfair... =====
Session token is: [skipped]
Logged in OK
===== Send 10 keepAlive requests =====
keepAlive result: OK duration 0.297
keepAlive result: OK duration 0.328
keepAlive result: OK duration 0.312
keepAlive result: OK duration 0.329
keepAlive result: OK duration 0.296
keepAlive result: OK duration 0.36
keepAlive result: OK duration 0.328
keepAlive result: OK duration 0.297
keepAlive result: OK duration 0.312
keepAlive result: OK duration 0.313
===== Logging out... =====
Logged out OK

Pretty long times isn't it?
The same with tuned HTTP:

===== Logging in to Betfair... =====
Session token is: [skipped]
Logged in OK
===== Send 10 keepAlive requests =====
keepAlive result: OK duration 0.079
keepAlive result: OK duration 0.078
keepAlive result: OK duration 0.078
keepAlive result: OK duration 0.094
keepAlive result: OK duration 0.078
keepAlive result: OK duration 0.078
keepAlive result: OK duration 0.078
keepAlive result: OK duration 0.078
keepAlive result: OK duration 0.078
keepAlive result: OK duration 0.078
===== Logging out... =====
Logged out OK

Much better isn't it? In the first example all the invocations are in separate HTTP connections (HTTPS connect is hell of slow). In the second the connection is single and no HTTPS handshake overhead is present.

After I resolve the connection pool issue (which is not much serious) it can be used in producation apps.

[elmariachi]

Originally Posted by AlgoTrader

The code works and works great. The first production app is being written. The first results are exellent, but there are several minor problems. All of them are certainly not the serouos problems.

The most important problem is to ensure the pool of persistent connections. After I resolve the connection pool issue (which is not much serious) it can be used in producation apps.

Thy for the heads up. Couple of questions:

• How many api calls per second do you think the js library will be able to handle (10, 20, 100, 200?) on what kind of machine?
• What is your mechanism of throttling to 20 requests/second (if requested) in order to avoid betfair charges?

[AlgoTrader]

Originally Posted by elmariachi

Thy for the heads up. Couple of questions:[*]How many api calls per second do you think the js library will be able to handle (10, 20, 100, 200?) on what kind of machine?

It depends on connection and request types. Simple test of sending 50 keepAlives using pool of 10 HTTPS connections:

===== Logging in to Betfair... =====
Session token is: [skipped]
Logged in OK
===== Send 50 keepAlive requests =====
keepAlive result: OK duration 0.328
keepAlive result: OK duration 0.328
keepAlive result: OK duration 0.328
keepAlive result: OK duration 0.328
keepAlive result: OK duration 0.344
keepAlive result: OK duration 0.344
keepAlive result: OK duration 0.344
keepAlive result: OK duration 0.359
keepAlive result: OK duration 0.344
keepAlive result: OK duration 0.109
keepAlive result: OK duration 0.187
keepAlive result: OK duration 0.281
keepAlive result: OK duration 0.359
keepAlive result: OK duration 0.406
keepAlive result: OK duration 0.406
keepAlive result: OK duration 0.406
keepAlive result: OK duration 0.406
keepAlive result: OK duration 0.422
keepAlive result: OK duration 0.422
keepAlive result: OK duration 0.422
keepAlive result: OK duration 0.422
keepAlive result: OK duration 0.437
keepAlive result: OK duration 0.453
keepAlive result: OK duration 0.484
keepAlive result: OK duration 0.484
keepAlive result: OK duration 0.484
keepAlive result: OK duration 0.484
keepAlive result: OK duration 0.484
keepAlive result: OK duration 0.5
keepAlive result: OK duration 0.5
keepAlive result: OK duration 0.5
keepAlive result: OK duration 0.515
keepAlive result: OK duration 0.531
keepAlive result: OK duration 0.562
keepAlive result: OK duration 0.578
keepAlive result: OK duration 0.562
keepAlive result: OK duration 0.562
keepAlive result: OK duration 0.562
keepAlive result: OK duration 0.578
keepAlive result: OK duration 0.578
keepAlive result: OK duration 0.578
keepAlive result: OK duration 0.594
keepAlive result: OK duration 0.609
keepAlive result: OK duration 0.64
keepAlive result: OK duration 0.64
keepAlive result: OK duration 0.64
keepAlive result: OK duration 0.64
keepAlive result: OK duration 0.656
keepAlive result: OK duration 0.656
keepAlive result: OK duration 0.656
===== Logging out... =====
Logged out OK

Please note that ALL the keepAlives were sent immediately, it is parrallel test

Everything is done in 0.656 seconds. I think 100-150 requests a second is not a problem using a good PC and a good Internet connection. Please note that sending some requests is subject to charges if more than 20 reqs/second is sent.

Originally Posted by elmariachi

[*]What is your mechanism of throttling to 20 requests/second (if requested) in order to avoid betfair charges?

There is no such mechanism for now. Some people want to block too many requests, some people got enough money to pay the charges.

It is application logic, only you knows what you want. That would not be a problem to make a "throttle" mode though

[elmariachi]

Originally Posted by AlgoTrader

It depends on connection and request types. Simple test of sending 50 keepAlives using pool of 10 HTTPS connections:
Please note that ALL the keepAlives were sent immediately, it is parrallel test

Everything is done in 0.656 seconds. I think 100-150 requests a second is not a problem using a good PC and a good Internet connection. Please note that sending some requests is subject to charges if more than 20 reqs/second is sent.

Hm. Sending the keep alives is a good start, but without a lot of processing (render larger responses into object and executing further logic) it might be difficult to talk about performance. I was more looking for answers in the regard how this type of processing performance compares to how the existing solutions (most interestingly c#).

Originally Posted by AlgoTrader

There is no such mechanism for now. Some people want to block too many requests, some people got enough money to pay the charges.

It is application logic, only you knows what you want. That would not be a problem to make a "throttle" mode though

Writing this logic is not trivial, since you need to account for the network delays etc., I would love to see this type of logic in the base lib, so not everybody has to role his own (but definitively as an option only). It could be set when creating the api instance, so that all calls via this instance are throttled for example. What do you think?

[AlgoTrader]

Originally Posted by elmariachi

Hm. Sending the keep alives is a good start, but without a lot of processing (render larger responses into object and executing further logic) it might be difficult to talk about performance. I was more looking for answers in the regard how this type of processing performance compares to how the existing solutions (most interestingly c#).

Network is always a bottleneck in BF applications. As to parsing performance - converting XML into JS object, especially for "compressed" calls - it is not an issue. Node.js is based on V8 JavaScript engine, it converts JavaScript into native code and it is blazingly fast. Not as Fast as C/C++ but the same order of magnitude. V8 is a hell of faster than PHP or Pyton

But you are right, there is a single thing to worry about. It is the garbage collector. Garbage collector may impose random delays in processing, but usually just a number of milliseconds.

Writing this logic is not trivial, since you need to account for the network delays etc., I would love to see this type of logic in the base lib, so not everybody has to role his own (but definitively as an option only). It could be set when creating the api instance, so that all calls via this instance are throttled for example. What do you think?

I think throttling in the lib is a worth option to do, but it is not the primary priority. The first priority is exellency in network work, minimal delays using reliable pool of connections. The second priority is betting emulator. Yes, I will embed the betting emulator so that there will be "training mode". Betting emulator is not that dificult to do, although it will be not so good in the whole range of prices.

[AlgoTrader]

There are words and there are facts. I modified the code to parse each invocation 100 times instead of doing it 1 time. 100 invocations a second is certainly enough for almost everybody. If not, you need a team of C++ pros and pay them good money.

            console.log("parse %s 100 times", self.action);
            console.time('duration');
            for(var cnt=0; cnt<100; ++cnt) {
                self.result = decoder.decode(self.xmlResponseBody, self.schema);
                decompress.decompressInvocation(self.action, self.result);
            }
            console.timeEnd('duration');

BTW - how beatiful parsing code is :-)

The results:

===== Logging in to Betfair... =====
parse login 100 times
duration: 62ms

===== Logging out... =====
parse logout 100 times
duration: 31ms

===== Get available tennis matches =====
parse getAllMarkets 100 times
duration: 570ms

===== Call getCompleteMarketPricesCompressed for marketId="105479661" =====
parse getCompleteMarketPricesCompressed 100 times
duration: 82ms

===== Call getMarketPricesCompressed for marketId="105479661" =====
parse getMarketPricesCompressed 100 times
duration: 42ms

The last three were quite heavy

Comments. XML parser is pure JavaScript - dom-js. Compressed calls parser is also pure JavaScript. I do not use any Node.JS modules that use C++ in order to ensure cross-platform compatibility. Betfair-sports-api works on Linux, Windows and MacOSX.

My CPU is Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
It's pretty usual desktop CPU

[AlgoTrader]

Today I released betfair-sports-api 0.1.3 that fixed network performance problems.

~/tmp> npm install betfair-sports-api
npm http GET https://registry.npmjs.org/betfair-sports-api
npm http 200 https://registry.npmjs.org/betfair-sports-api
npm http GET https://registry.npmjs.org/dom-js
npm http GET https://registry.npmjs.org/async
npm http 200 https://registry.npmjs.org/dom-js
npm http GET https://registry.npmjs.org/dom-js/-/dom-js-0.0.7.tgz
npm http 304 https://registry.npmjs.org/async
npm http 200 https://registry.npmjs.org/dom-js/-/dom-js-0.0.7.tgz
npm http GET https://registry.npmjs.org/sax
npm http 200 https://registry.npmjs.org/sax
betfair-sports-api@0.1.3 ./node_modules/betfair-sports-api 
├── async@0.1.18
└── dom-js@0.0.7

~/tmp> node node_modules/betfair-sports-api/test/keep_alive_serial.js 
===== Logging in to Betfair... =====
Session token is: [skipped]
Logged in OK
===== Send 20 serial keepAlive requests =====
keepAlive result: OK duration 0.285
keepAlive result: OK duration 0.083
keepAlive result: OK duration 0.081
keepAlive result: OK duration 0.079
keepAlive result: OK duration 0.073
keepAlive result: OK duration 0.074
keepAlive result: OK duration 0.07
keepAlive result: OK duration 0.072
keepAlive result: OK duration 0.071
keepAlive result: OK duration 0.074
keepAlive result: OK duration 0.073
keepAlive result: OK duration 0.071
keepAlive result: OK duration 0.073
keepAlive result: OK duration 0.072
keepAlive result: OK duration 0.074
keepAlive result: OK duration 0.086
keepAlive result: OK duration 0.072
keepAlive result: OK duration 0.071
keepAlive result: OK duration 0.071
keepAlive result: OK duration 0.069
===== Logging out... =====
Logged out OK
~/tmp>

First keepalive is 285 ms, all the next are just ~80ms as they reuse already established HTTPS connection. I am located in Moscow, it is 2500 km

[AlgoTrader]

There is one more highly desired feature implemented: gzip compression

1) This highly reduce network traffic. That's great for hostings and slow internet connections. Traffic is reduced in times, as BF XMLs are compressed very good.

2) This highly reduce latencies of 'big' responses. Especially good for getAllMarkets, which can be easily compressed by 90%.

betfair-sports-api most important goal is to achive exellency in network operations to provide exellent latency and high throughput of parrallel invocations. The goals seem to be achived.

So the results.
Gzip compression disabled:

~/betfair-sports-api/test> node get_market.js 
===== Logging in to Betfair... =====
login response: raw length=982, xml length=982, compression=0%
Session token is: [skipped]
Logged in OK
===== Get available tennis matches =====
getAllMarkets response: raw length=91908, xml length=91908, compression=0%
action: getAllMarkets error: null duration: 0.552
===== Call getMarket for marketId="105530637" =====
getMarket response: raw length=4719, xml length=4711, compression=0%
action: getMarket error: null duration: 0.088
marketId: 105530637
market name: Match Odds
market time: Tue, 24 Apr 2012 09:00:00 GMT
	playerOneId: 5229875
	playerOneName: Guillermo Garcia Lopez
	playerTwoId: 2257305
	playerTwoName: Olivier Rochus
===== Logging out... =====
logout response: raw length=813, xml length=813, compression=0%
Logged out OK
~/betfair-sports-api/test>

Gzip compression enabled:

~/betfair-sports-api/test>  node get_market.js 
===== Logging in to Betfair... =====
login response: raw length=481, xml length=982, compression=51%
Session token is: [skipped]
Logged in OK
===== Get available tennis matches =====
getAllMarkets response: raw length=8755, xml length=91908, compression=90%
action: getAllMarkets error: null duration: 0.134
===== Call getMarket for marketId="105530637" =====
getMarket response: raw length=1803, xml length=4711, compression=62%
action: getMarket error: null duration: 0.09
marketId: 105530637
market name: Match Odds
market time: Tue, 24 Apr 2012 09:00:00 GMT
	playerOneId: 5229875
	playerOneName: Guillermo Garcia Lopez
	playerTwoId: 2257305
	playerTwoName: Olivier Rochus
===== Logging out... =====
logout response: raw length=393, xml length=813, compression=52%
Logged out OK
~/betfair-sports-api/test>

login compression = 51%
getAllMarkets compression = 90%
getMarket compression = 62%
logout compression = 52%

getAllMarkets response is huge so compression makes HTTP request much faster and smaller in size!