Betfair Developers


PHP
Discussions about the widely-used general-purpose scripting language PHP that is especially suited for Web development and can be embedded into HTML.

Post Reply
Fred77 Fred77 is offline
Senior Member
 
Default Streaming and stuff

After a break from botting I'm back and currently having a go at consuming the stream api with PHP.

Just wondering how other people have gone about it, or how you think you'd go about it. Interested in hearing any PHP waffle.




I've reached the stage where I can connect and reliably start streaming using PHPs stream functions in a reasonably asyncronous manner and build simple market representations. Seeing the data in full flow for the first time was like viewing the matrix! I have puzzled over how to interface this code with bot code and what the overall structure / program flow would be like.

My streaming code is based on a stream_select() loop, whereas my existing APING code uses a curl_multi_select() loop where necessary. These cannot be easily combined (without an ugly hack) as phpcurl cannot currently import/expose underlying sockets, phpcurl doesn't support streaming either.

My current idea and goal is rather than integrate streaming code into existing bots, is to keep my streaming code as a separate process, like a local server that can distribute data to multiple strategy scripts. I think that will leave my options reasonably open, it can serve strategies that want constant updates as well as ones that only need slower updates.

To build my local server I think an event loop is the way to go, so that I can asynchronously:
  • resolve dns (not essential as betfair rarely changes IP addresses)
  • access the stream
  • access session functions login/keepalive
  • perhaps access APING functions, needed to get any missing information, e.g. runner names
  • serve local bots using IPC, probably unix sockets
  • log what's going on
  • store market data for testing purposes

Starting to sound complicated! I believe the above would be relatively straightforward if one was familiar with the right extensions/libraries/packages. I don't know which library would do it all and do it well.

I have been experimenting and have found it would be possible to do everything in vanilla php using non-blocking stream functions and stream_select() rather than an event loop, but there's a few wheels that need reinventing in order to get desired performance/efficiency.

SSL is a complication too. When it's working it's fine, but establishing connections, slow connections, and closing connections, are problem areas for the stream functions. Establishing an SSL connection can be realised asynchronously with a slight workaround. With non-blocking streams slow connections result in a lot of "" empty reads between valid reads, even though stream_select indicates data. Closed connections also result in "" empty reads too, no easy way to differentiate between slow & closed. I made an ugly hack to extract the underlying FD from a stream, which fopen would DUP so that it could then be checked for EOF, this seems to work.

I'm currently experimenting with pecl event extension, it has an event loop and it offers an SSL integration that seems to work better, I haven't tried it in earnest yet though.
Fred77 Fred77 is offline
Senior Member
 
Default

All quiet in PHP land. Development of my local stream distributing server as detailed above is proceeding rather slowly. Event driven programming is proving more complicated than I anticipated - I bit off more than I can chew! Although I've used PHP for 15 years I've never become fully proficient.

After googling around I see react PHP library getting lots of mentions, looks like it has all the components to do everything, it doesn't require any extensions either. It's based on stream_select() but with lots of bits added in order to simplify event programming.

As another diversion I checked out the php pthreads extension. I thought it looked sexy, but after playing with it for a while I couldn't see it offering much benefit over a non-blocking event loop, especially when my target machine will likely be a single core VPS.

Is anybody else still using the php hammer?
LiamP LiamP is offline
Member
 
Default

Why a separate server for streaming?

I use python so maybe PHP design patterns differ but using my library processing streaming data is not very cpu intensive, I use multiple AWS nano instances (0.5gb ram) My framework then follows an event driven architecture, similar to flumine, processing any market update and order updates as they come.
Fred77 Fred77 is offline
Senior Member
 
Default

Quote:
Originally Posted by LiamP View Post
Why a separate server for streaming?
My main reason is so that multiple standalone bots can consume the same, or subsets of the same data.

If any bots crash they won't take the rest down. An alternative is multiple strategies within one program, but that doesn't suit my php development style (start/stop/debug/start/crash/debug etc!)

I'll no doubt want a few different streams too. I don't know how many streams and subscriptions you're allowed, so better to be conservative from the outset - also further charges might be brought in etc.

I don't know about design patterns, I think these days you can probably coerce many languages to use any pattern. This is my first experience of fully event driven programming so I'm having to (re)learn a few things. I agree, processing streaming data is not cpu intensive, it looks much easier to do in python, plus python's threading looks positively dreamy. But I've enough distractions - I'm sticking with PHP lol
Fred77 Fred77 is offline
Senior Member
 
Default

I'm unfamiliar with how the stream operates during error conditions, would be nice to get a few pointers. My usual method of finding out these things is just to leave whatever I'm testing running 24/7 to see what happens. This morning when betting went down the stream didn't stop, there were some infrequent market updates but mostly the stream kept ticking with HEARTBEATs.

An inital measure I've just started is to record the number of market updates received per x seconds, naturally this will only provide a health indication for busy streams.
LiamP LiamP is offline
Member
 
Default

Quote:
Originally Posted by Fred77 View Post
I'm unfamiliar with how the stream operates during error conditions, would be nice to get a few pointers. My usual method of finding out these things is just to leave whatever I'm testing running 24/7 to see what happens. This morning when betting went down the stream didn't stop, there were some infrequent market updates but mostly the stream kept ticking with HEARTBEATs.

An inital measure I've just started is to record the number of market updates received per x seconds, naturally this will only provide a health indication for busy streams.
Interesting, I thought the issue this morning was to do with bet placement? I had no issues streaming data from AUS racing, I assume any issues with streaming and you will get a timeout error due to not receiving a heartbeat (I have mine set to 11s)
Fred77 Fred77 is offline
Senior Member
 
Default

Yes I think it was a bet placement problem, I presume as no bets were getting through then there were no market updates as reflected by the quiet stream. I was only subscribed to UK and US racing markets so don't know if AUS went quiet too. I've got my timeout at 10 seconds and I currently get several timeouts per day on my home connection, that's not too bad.
Post Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump