Getting Equity Derivatives Watch table from NSE India using Selenium and BeautifulSoup

The goal of this project is to automate the process of exporting Equity Derivatives Watch table from NSE into excel/CSV. To do this, we will be using selenium and BeautifulSoup. The process is as follows:

  1. Use Selenium to visit NSE website and hover over the Live Market section. Then click on "Equity Derivates" option to see the table.
  2. Use BeautifulSoup to extract the table contents.
  3. Use Pandas to convert the contents of the table into a data frame.
  4. Export the final table into CSV.

Desired Result:

NSE-Table.png

Importing necessary packages

In [1]:
import pandas as pd

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains

Visiting NSE website

In [2]:
url = "https://www.nseindia.com/"
driver = webdriver.Chrome()
driver.get(url)
import time
time.sleep(5)

Hover over the Live Market section on NSE website

In [3]:
action = ActionChains(driver)

live_market_section = driver.find_element_by_xpath('//*[@id="main_livemkt"]/a')
action.move_to_element(live_market_section).perform()

Clicking on Equity Derivates option under Live Market

In [4]:
equity_derivatives = driver.find_element_by_xpath('//*[@id="main_livewth_deri"]/a')
equity_derivatives.click()

Pushing HTML into BeautifulSoup

In [5]:
html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
#soup = BeautifulSoup(html,'lxml')
time.sleep(7)

Finding our table on the page

In [6]:
result = soup.find('table',{'class': 'tablesorter'})
table_rows = result.find_all('tr')

Converting data into a Table using pandas

In [7]:
l = []

for tr in table_rows:
    td = tr.find_all('td')
    row = [tr.text for tr in td]
    l.append(row)
    
    
# Creating the table using pandas
equity_derivatives_table = pd.DataFrame(l, columns=["InstrumentType","Underlying","ExpiryDate","OptionType",
                                 "Strike Price","PrevClose","OpenPrice","HighPrice","LowPrice",
                                 "LastPrice","No. of Contracts Traded","Turnover (lacs)",
                                 "Premium turnover (lacs)","Underlying Value"])

equity_derivatives_table = equity_derivatives_table.drop([0], axis=0)

equity_derivatives_table
Out[7]:
InstrumentType Underlying ExpiryDate OptionType Strike Price PrevClose OpenPrice HighPrice LowPrice LastPrice No. of Contracts Traded Turnover (lacs) Premium turnover (lacs) Underlying Value
1 Index Options NIFTY 14AUG2019 CE 11,200.00 32.85 43.00 66.70 32.00 38.65 4,10,207 34,60,413.96 14,675.16 11109.65
2 Index Options NIFTY 14AUG2019 PE 11,000.00 65.15 64.65 64.65 25.95 31.80 3,80,932 31,53,002.73 10,313.73 11109.65
3 Index Options BANKNIFTY 14AUG2019 PE 28,000.00 184.35 160.40 160.40 74.15 79.00 5,50,346 30,93,316.55 11,378.95 28431.90
4 Index Options NIFTY 14AUG2019 PE 11,100.00 115.00 105.00 108.30 48.05 64.50 3,27,806 27,45,147.42 16,162.47 11109.65
5 Index Options NIFTY 14AUG2019 CE 11,100.00 60.20 86.75 120.00 70.20 76.00 3,25,215 27,29,020.53 21,605.66 11109.65
6 Index Options BANKNIFTY 14AUG2019 CE 28,500.00 133.25 245.00 266.00 160.55 199.00 4,38,770 25,19,252.36 18,263.36 28431.90
7 Index Options BANKNIFTY 14AUG2019 CE 29,000.00 43.00 52.00 88.00 43.20 53.90 3,95,433 22,98,303.26 4,791.86 28431.90
8 Index Options NIFTY 14AUG2019 CE 11,300.00 15.80 20.90 34.45 15.30 16.20 2,64,860 22,49,187.81 4,499.31 11109.65
9 Index Options BANKNIFTY 14AUG2019 CE 28,700.00 83.85 120.00 167.00 100.00 121.00 3,42,279 19,73,808.67 9,127.21 28431.90
10 Index Options BANKNIFTY 14AUG2019 CE 28,600.00 103.95 137.00 210.00 129.75 153.50 3,38,895 19,49,942.86 11,463.46 28431.90
11 Index Options BANKNIFTY 14AUG2019 PE 28,200.00 260.65 199.85 236.60 104.25 120.60 3,29,679 18,69,548.29 10,158.73 28431.90
12 Index Options NIFTY 14AUG2019 CE 11,150.00 44.45 59.70 90.00 50.70 55.00 1,89,224 15,91,818.99 9,433.29 11109.65
13 Index Options BANKNIFTY 14AUG2019 CE 28,800.00 64.90 97.00 130.00 76.00 94.40 2,68,312 15,51,005.42 5,528.30 28431.90
14 Index Options NIFTY 14AUG2019 PE 10,900.00 37.35 28.00 30.00 15.35 16.50 1,86,951 15,31,134.30 2,809.87 11109.65
15 Index Options NIFTY 14AUG2019 CE 11,400.00 8.10 9.55 15.60 5.30 5.80 1,63,906 14,02,592.40 1,196.10 11109.65
16 Index Options BANKNIFTY 14AUG2019 PE 28,300.00 313.40 257.20 283.80 134.50 150.00 2,44,044 13,90,624.70 9,335.66 28431.90
17 Index Futures NIFTY 29AUG2019 - - 11,063.95 11,108.00 11,186.80 11,068.25 11,119.05 1,60,680 13,41,712.55 - 11109.65
18 Index Options BANKNIFTY 14AUG2019 PE 28,100.00 219.15 166.25 193.25 90.00 96.50 2,37,229 13,39,185.70 5,958.72 28431.90
19 Index Options BANKNIFTY 14AUG2019 CE 28,400.00 161.30 210.00 323.35 199.45 251.00 2,24,819 12,88,305.50 11,333.58 28431.90
20 Index Options BANKNIFTY 14AUG2019 PE 27,900.00 151.20 107.85 124.85 60.00 62.15 2,13,625 11,95,693.73 3,666.23 28431.90

Exporting table into CSV

In [8]:
equity_derivatives_table.to_csv("Equity Derivates Watch Table from NSE.csv", index=0)

There is an alternate (and easier) way to get the data using the request library and the final url of the table and then using pandas but I wanted to make use of selenium package.

Author: Amandeep Saluja