Scraping iMessage and Messenger Messages and Displaying with Vue Frontend

Credit: she founded the project and provided the first version of the scraper.

A while ago my partner in the organization started message-analyzer because she thought it would be interesting to analyze the message data between us. She managed to scrape text messages out of both iMessage and Messenger (the two chat softwares that we use), put them together, built something that could decide which one of us a messaging is coming from. I believe the highest success rate she got to was 86%.

I was looking around in the project after she got most of it done and noticed this file called app.py that runs a Flask application and serves the text messages on a web server. Since I’m pretty much a frontend developer now (no), I came up with the idea of displaying all of our messages on a web page, hopefully merging contents on both Apple and Facebook platforms.

iMessage

I started with iMessage. It wasn’t too hard to simply take the output of the function that she wrote and serve it over the api endpoint. For the frontend I decided to try out Vue.

It wasn’t long before I got to the following:

The main component simply requests all messages and pass each data to a Message component. I added pagination for some convenience.

Message component looks like this:

It just displays the message content. If hovered, the delivered time is shown as a tooltip.

It all looked good, but how about attachments? There were hundreds of interesting images, stickers and files that we sent each other. It would not be as interesting if those were lost for the web page.

To show attachments, I dug deeper into how Apple stores messages.

Inspired by my partner, Apple stores messages in a sqlite database located in ~/Library/Messages/chat.db, so I took the liberty of looking at the schema.

Three tables caught my attention: attachment, message_attachment_join, and message.

attachment:
    filename
message:
    ROWID
message_attachment_join
    message_id
    attachment_id

The message_id matches with the ROWID on the message table. filename is actually a path to the attachment file on the local machine. With these information at hand, I revised the sqlite query to

After the messages and attachments are selected, I served the attachments over the api endpoint ‘/attachments’, and voila pictures on the page!

I later also displayed reactions to messages but I’d like to get to scraping Messenger soon.

Messenger

Scraping Messenger is a little more tricky: my partner did it by scrolling up all the way to the top, saving the html file and extracting information from there. However, since the data is parsed once already by the Messenger frontend, it was a little difficult to get the dates and attachments as well as the messages.

I went into Chrome devtools and saw that the juicy request was to the url facebook.com/graphqlbatch. Ah so they use their own product. What’s frustrating is that each request at most retrieves ~200 messages, and Chrome doesn’t let me copy multiple request responses at a time.

I tried to reverse engineer how the requests are formatted, but was stuck at figuring out how the message count offset was sent. I came to the idea of writing a Chrome extension to capture the web requests.

The only API that allows you access to response bodies is devtools. Creating an extension is also easy – just need to have a manifest.json file that specifies the extension and some js scripts to be run by the browser, so I did this:

and used pyauthogui from my partner’s code to automatically scroll up like an idiot. I was able to get all messages in the devtools window of the devtools window (no typo). The repository is here.

All that was left was parsing the data retrieved and making sure both message sources end up having the same format when returned by the Flask server. Messenger had more attachment types and multiple attachments so it took me longer.

Due to privacy reasons, I can’t do a demo here :/ well mostly it’s just that I’m too lazy to put up a page with fake message data.

For future features I plan to do searching, improve pagination, style the Messenger system messages (“you waved at each other”), and make the UI prettier and easier to use.

63 thoughts on “Scraping iMessage and Messenger Messages and Displaying with Vue Frontend”

  1. Thank you for ɑny other wonderfuⅼ article.

    Wһere else may anyone get that type of info in such an ideɑl way
    of writing? I have a presentation next week, and I am on the lok ffor such information.

  2. Every guy out there needs to talk to a hot girl sometimes. There’s just one site that’s full of the hottest babes on the entire internet. That site is http://www.camgirl.pw and it’s going to put a smile on your face. Make sure you spend some of your precious free time there. It’ll be the best decision that you’ve made in quite some time.

  3. I?m impressed, I must say. Really rarely do I encounter a blog that?s both educative and entertaining, and let me tell you, you have hit the nail on the head. Your idea is outstanding; the issue is something that not enough people are speaking intelligently about. I am very happy that I stumbled across this in my search for something relating to this.

  4. Real men like to talk to sexy girls. There’s no denying that. Where does a man go to talk to a sexy girl? There’s just one place on the the internet to do that. The hottest girls can be found at http://www.camgirl.pw Have yourself a total blast and meet a few sexy girls. You’ll have a whole lot of fun doing so.

  5. I really like your blog.. very nice colors & theme. Did you make
    this website yourself or did you hire someone
    to do it for you? Plz respond as I’m looking to create mmy own blog and would like
    to know where u got this from. many thanks

  6. andrew strutt DOB 6/3/1975 sprint phone 312-917-5986 google voice 702-514-0650 works at leidos cyber has raped 2 girls at defcon hacker conventions. andy strutt wife Sichan Li is in the United States illegally. andrew david strutt has an undiagnosed psychiatric disorder from a head injury. andrew strutt has taken money from Lebanse Intelligence Officer and recently arrested hacker and owner of Krypton Security Khalil Sehnaoui. andrew david strutt accepted a hotel suite in Las Vegas in 2015, 2016, and 2017 in excess of $2,000 a night from Khalil Sehnaoui. andy strutt wife is a former reporter and Chinese Communist Party reporter. r0d3nt wife has access to his secret material.

  7. I kind of agree with everything said here. There’s just one problem. It doesn’t talk about what’s the best site to join to get laid. That’s the most important part of it all. If you’re wanting to get laid, then this is the site for you https://t.irtyf.com/hkz06n8e68?offer_id=3785&aff_id=29696&nopop=1 You’re going to get some pussy there. Don’t beat off tonight. Get yourself some action. You can beat off tomorrow. Today is your day to get some pussy.

  8. Sick and tired of being bored? You’ll never be bored again. Not when you have http://www.camgirl.pw at the top of your bookmarks. This site is full of hot girls who know how to have a good time. You’ll have to pick your tongue up off of the floor while watching these beauties in action.

  9. You’ve been a busy cowboy lately. Don’t you think it’s time to settle down and relax? You can do just that with the babes over at http://www.camgirl.pw There’s plenty of hot girls over there who know how to treat a cowboy right. Take off your spurs and mosey on into a chat with one of these beauties.

  10. You should set aside time each day to talk to a hot girl. There are plenty of them over at http://www.camgirl.pw You’re really going to have yourself a good time there. It’s wall to wall babes and that’s just the beginning. Check it out and get ready to smile.

  11. You deserve a break. You’ve been working hard this week. All that hard work has stressed you out. The best way to get rid of the stress is by having a little fun with a cam girl. There’s a whole lot of them over at http://www.camgirl.pw The babes are never ending when you visit that site. You’re in for a real treat the very second your eyes catch a glimpse of this.

  12. Hello there, I found your web site by the uuse of Google while lookng for a comparable subject, your site
    came up, it appears tto be like great. I’vebookmarked it in my google
    bookmarks.
    Hi there, just changed nto alert to your weblog via
    Google, and found tat iit is truly informative. I am going tto
    be careful for brussels. I’ll appreciate should you continue this in future.
    Lots of people shall be benefited out of your writing.
    Cheers!

  13. Thank you, I’ve recently been searching for
    information about thhis subjrct for a long time and yours is the greatest I’ve came upon so far.
    But, what about the bottom line? Are you positive concerning the source?

  14. Greetings, There’s nno doubt that your site could possibly be having internet
    browser compatibility problems. Whenver I take a look at your site in Safari, it looks fine but when opening in IE, it’s
    got some overlapping issues. I sjmply wanted to provide you withh a quick heads up!
    Apart frm that, wonderful site!

  15. Wow, wonderful weblog layout! Howw lengthy have you ever been running a bllog for?
    yyou made blogging look easy. The full glance of your website is fantastic, let alone the content!

  16. Have you ever thought abhout including a little bit more than just your articles?
    I mean, what you sayy is valuable and everything.

    Nevertheless just imagine if you added some ggreat graphics or videos to give your posts more, “pop”!
    Your content is excellent but witth images and clips, this site could definitely be one of thee greatest in its field.
    Wonderful blog!

  17. Do you mind if I quote a couple of your articles as long as I provide credit and sources back to your website? My blog site is in the exact same niche as yours and my visitors would certainly benefit from a lot of the information you present here. Please let me know if this alright with you. Regards!

  18. When someone writes an paragraph he/she keeps the plan of a user in his/her brain that how a user
    can know it. Therefore that’s why this article is great.
    Thanks!

  19. Having read this I thought it was extremely enlightening.

    I appreciate you spending some time and effort to put this contentt together.
    I once again find myself personally spending a lot of time both reading and commenting.
    But so what, it was still worth it!

  20. Good day I am so excited I found your webpage, I really found you
    by accident, while I was looking on Yahoo for something
    else, Anyhow I am here now and would just like to say thanks a lot
    for a marvelous post and a all round thrilling blog (I also love the theme/design), I don’t
    have time to go through it all at the moment
    but I have saved it and also included your RSS feeds, so when I have time I will be back to read a lot more, Please do keep up the
    awesome job.

  21. Ahaa, its pleasant dialogue concerning this paragraph here at this website, I
    have read all that, so at this time me also commenting at this place.

  22. Have you ever thought about including a little bit more than just
    your articles? I mean, what you say is important and all.
    However just imagine if you added some great photos or videos to give your posts more,
    “pop”! Your content is excellent but with pics and video clips, this site could undeniably be one of the
    greatest in its field. Excellent blog!

  23. dissertation
    essay on health
    creative writing essays
    essay transitions
    send your resume
    academic essay writers

    custom essay writing
    igcse english cambridge
    gcse chemistry coursework
    christmas essay
    write papers for money
    how to write an essay

    write my paper for me
    write a cv in minutes
    i gcse english
    writing of cv
    gcse syllabus
    how can i do a resume

    custom essay writing service
    buy papers
    education essay
    cv new zealand
    edexcel igcse
    environmental management igcse

    essay writing
    writing essay online
    get essay written
    gcse online courses
    work from home
    i gcse english

    write my essay
    custom writing uk
    writing a thesis for a research paper
    cv centre
    online writing companies
    write cv

  24. thesis statement
    how to write professional resume
    cambridge igcse syllabus
    creating your resume
    examples of research papers
    classic essays

    type my essay
    music igcse
    1000 word essay
    what is a resume
    course to study
    any essay

    dissertation writer
    essay writing prompts
    essay about love
    preparing a resume
    travel essay
    igcse logo

    write my essay
    essay on sports
    writing work from home
    edexcel igcse subjects
    educational opportunities definition
    resume writing information

    custom writing service
    gcse art coursework
    essay plan
    best resume writing
    will writing service
    assignment help uk

    best custom writing
    english igcse
    gcse maths syllabus
    how make a curriculum vitae
    write essays online
    about me essay

  25. custom writing service
    letter writing service
    best term paper
    help on writing essays
    igcse edexcel physics
    resumes cover letters

    thesis paper
    igcse india
    law coursework
    expert writing
    igcse hindi
    academic writing service

    essay writing
    essay of pollution
    professional cv
    gcse sociology coursework
    how is a cv written
    water pollution essay

    thesis statement
    best resume writing
    apa style essay
    professional cv writing service uk
    writing a research paper
    how to write covering letter

    write my paper
    what is resume writing
    writing a good cv
    college research papers
    free essay download
    online help chat

    essay writing
    research papers
    gcse home learning
    will writing
    extended essay topics
    biology syllabus igcse

  26. Great blog here! Allso yolur site loads up fast!
    What host are you using? Can I get your affiliate
    link to your host? I wish my web site loaded up as quickly
    as yours lol

  27. Link exchange is nothing else except it iss only placing the other person’s web site link on your
    page at appropriate place and other person will also do similar
    in favor of you.

  28. I have been exploring for a bit for any high-quality
    articles or blog posts in this kind of house . Exploring in Yahoo
    I finally stumbled upon this site. Studying this info So i’m satisfied to express that I’ve
    a very just right uncanny feeling I discovered just what I needed.
    I so much unquestionably will make certain to do not omit
    this web site and provides it a look on a constant
    basis.

  29. Howdy! This blog post could not be written any better!

    Going through this post reminds me of my previous roommate!
    He always kept preaching about this. I am going to forward this article to him.
    Pretty sure he’s going to have a good read.
    Many thanks for sharing!

  30. Hey there, I think your site might be having browser compatibility issues.
    When I look at your blog in Firefox, it looks fine but when opening in Internet Explorer, it has some overlapping.

    I just wanted to give you a quick heads up! Other
    then that, awesome blog!

Leave a Reply

Your email address will not be published. Required fields are marked *