Thread: Getting high-frequency, real-time stock index data

  1. #1
    Registered User
    Join Date
    Mar 2016
    Posts
    203

    Getting high-frequency, real-time stock index data

    I'd like to get the level of the Dow Jones index every 2 minutes b/w 9.30 ET and 16.00 ET (i.e from market open to market close). I particularly like the yahoo finance page: ^DJI : Summary for Dow Jones Industrial Average - Yahoo Finance (in this particular case the number captured would be 23,271.28 i.e. 11/15's close at the time of writing) but can consider other public pages if that helps.
    The idea is to read the index level every two minutes and append it to a .txt file. I was considering using a std::thread for this task and then detaching it so that it continues to run in the background but am not sure how to read in the actual data first. In addition to native C++ am able to use the boost library. If someone could give me an outline how to proceed that'd be much appreciated. Many thanks,
    Sean

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    Assuming you have permission to scrape the site every 2 mins, do you really need to do this in C++?
    There are plenty of other ways which are far easier to do than cranking out a lot of C++.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Guest
    Guest
    Also, does the program do something else in the meantime, or why do you want to spawn (and detach) a separate thread? If you just want to do something every N minutes and keep the program running forever, you should consider a simple loop and timer instead.

  4. #4
    Registered User
    Join Date
    Mar 2016
    Posts
    203
    ... do you really need to do this in C++?
    no, not really, it's just my programming 'skills' are largely limited to C++ so I was hoping to work within that.
    There are plenty of other ways which are far easier to do than cranking out a lot of C++.
    I know this is a C++ forum but if someone could drop me a line (either publicly here or by private message) with a couple of options in this regard I'd be very grateful.
    Thanks for your replies, Sean

  5. #5
    Registered User
    Join Date
    Dec 2010
    Location
    Trinidad, CO (log cabin in middle of nowhere)
    Posts
    148
    My first impression when I saw your post about this was that somewhere in my distant past I seem to recall encountering a Microsoft ActiveX Control that did this. A search revealed this...

    The Stock Ticker Control

    But I have other recollections of this. If I was attempting to do this, I believe that's where I'd start. If you are working with MS products, it might be a start. It does involve COM, which most C++ coders don't seem to like, it seems.

  6. #6
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    A few mins, and a few lines of perl (link)
    Code:
    $ cat fetcher.pl 
    #!/usr/bin/perl
    use warnings;
    use strict;
    use HTML::TreeBuilder 5 -weak;
    use WWW::Mechanize;
    
    my $domain = "https://finance.yahoo.com/quote/%5EDJI?p=%5EDJI";
    my $mech = WWW::Mechanize->new();
    $mech->get($domain);
    my $tree = HTML::TreeBuilder->new;
    $tree->parse_content($mech->content);
    $tree->elementify();
    my $price = $tree->look_down("_tag"=>"div","id"=>"quote-header-info");
    if ( $price ) {
      print $price->as_trimmed_text() . "\n";
    } else {
      print "Can't find it\n";
    }
    $ ./fetcher.pl 
    ^DJI - Dow Jones Industrial AverageDJI - DJI Real Time Price. Currency in USD23,358.24-100.12 (-0.43%)At close: 4:52PM EST
    Feel free to tweak the look_down to locate the specific bit of HTML which interests you.

    > no, not really, it's just my programming 'skills' are largely limited to C++ so I was hoping to work within that.
    When the only tool in the toolbox is a hammer, pretty soon all the problems look like nails.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  7. #7
    Registered User
    Join Date
    Mar 2016
    Posts
    203
    Thanks for the link, code and the nudge to upgrade my skills - the last is particularly apposite and I've been looking at R for solving this (and other) issue(s) and seem to be making some progress. Cheers, Sean

  8. #8
    Registered User
    Join Date
    Dec 2010
    Location
    Trinidad, CO (log cabin in middle of nowhere)
    Posts
    148
    It was so long ago...

    Did some searching of some old interactions I had on another forum about this and came up with this...

    Client APIs | CQG, Inc.

    Seems to me I was helping a fellow convert some Visual Basic 6 COM code to PowerBASIC, and he had an ActiveX dll that produced real time stock market data. The CQG object/dll/library was something one purchased if I recall, but if I recall correctly there was a free download of a trial version.

  9. #9
    Registered User
    Join Date
    Mar 2016
    Posts
    203
    I started, so I'll finish (and in case anyone's interested), using R: library(rvest) webpage <- data.frame("https://www.bloomberg.com/quote/INDU:IND" - Pastebin.com
    Salem - your sample code really helped me a lot in terms of where and how to look, thank you very much indeed

  10. #10
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    Nice
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. How to transfer audio data in real time?
    By leetow2003 in forum Networking/Device Communication
    Replies: 6
    Last Post: 02-21-2012, 03:10 PM
  2. Host/Server real time data transfer
    By dannyrod01 in forum Networking/Device Communication
    Replies: 9
    Last Post: 05-25-2011, 11:38 PM
  3. Stock Data
    By bskaer1 in forum C++ Programming
    Replies: 0
    Last Post: 10-16-2010, 11:59 PM
  4. Niche Question: real time stock quotes
    By keira in forum C Programming
    Replies: 0
    Last Post: 03-08-2008, 11:06 AM
  5. Real Time?
    By Sebastiani in forum C Programming
    Replies: 1
    Last Post: 09-23-2001, 03:33 PM

Tags for this Thread