PghFree

For my second project in General Assembly’s Web Development Immersive I decided to take on a project which combined my two passions, music and coding. I developed an application called PghFree. The goal for this project was to create a centralized database with all of the free music events happening around the Pittsburgh area in hopes to bring the community of music lovers in Pittsburgh a bit closer together.

To gather information on events that had already been published on the web, I used a Web crawler written in Python called Scrapy. I created five “spiders” that “crawled” each website I specified and gathered information about free music events. With this data as my seed data, I created a backend for my application which utilizes RESTful API conventions. In addition to events I published on the page from my research, users have the ability to login and post events they are hosting as well! Using Google Map’s geolocation API, I added a maps feature to my application which allows users to see a dynamically created map of the specific location for each event and to find directions.

Example of a Scrapy Spider used to crawl eventbrite.com:

import scrapy




class EventbriteSpider(scrapy.Spider):
    name = "firstCrawler"


    def start_requests(self):
        urls = [
            'https://www.eventbrite.com/d/pa--pittsburgh/free--music--events/free/?page=1'
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)


    def parse(self, response):
    	for data in response.css('.search-event-card-wrapper'):
            yield {
                'title': data.css('.card-text--truncated__three::text').extract_first(),
				'time': data.css('.eds-l-mar-top-1::text').extract_first(),
                'location': data.css('.card-text--truncated__one::text').extract(),
                'findOutMore': data.css('.eds-media-card-content__primary-content a::attr(href)').extract(),
                'eventPhoto': data.css('.eds-media-card-content__image').xpath('@src').extract(),
              }

This project uses a non-relational database called mongoDB and is deployed online using heroku. In addition to Full CRUD (create, read, update, destroy) functionality. I wanted to provide the user with a dynamic user interface, to do this I utilized the Bootstrap framework along with JQuery to add a pagination feature.

The long term goal for this project is all about automation. As of right now I have to manually run my web crawler in order to add data to my database. I would like to automate this process so that the app would in a way become self-sustaining.It was an amazing experience building this application from the bottom up. My biggest achievement has to be the use of Python. Before this project I had no experience coding in python so to teach myself enough to construct my web crawler was a major win for me!

Follow this link to the live app!