Most people first hear about EB-5 as a route to a U.S. green card through investment. The route is real, but it comes with ongoing obligations, a capped supply of visas, and a process that runs through several government bodies.

A green card comes with conditions

A green card is not a document you collect and forget. Permanent residents are taxed on their worldwide income from the day they receive permanent residence, so families with income abroad need tax planning from the outset.

The U.S. also expects green card holders to make the country their primary home. You can show this through a home you own or rent, along with everyday ties such as bank accounts, investments, credit cards and a driver’s license. Stay abroad for more than six months and you may face questions at the border about whether you have given up your residence. The guide recommends a re-entry permit from USCIS, which lets a holder remain outside the U.S. for up to a year without losing that status.

Male green card holders also have a duty to register with Selective Service between the ages of 18 and 25. Registration does not mean serving in the military. It does mean a registrant could be called up if the government ever chose to draft men.

Only 9,940 EB-5 visas a year

Each fiscal year, which runs from October to September, 9,940 EB-5 visas are available. That figure counts investors, spouses and children individually. Within it, 20% are reserved for rural projects, 10% for high-unemployment areas and 2% for infrastructure.

Unused visas in a reserved category do not disappear. They roll over to the same category the following year, then join the general pool from the third year. The guide also notes that the true number can be somewhat higher in some years, because unused family-based green cards spill over into employment categories, and EB-5 can claim a share of that spillover.

The bigger point is demand. When you count each applicant’s spouse and children, demand has outstripped supply in many years. That creates queues, and the guide promises a full discussion of queuing and allocation later in Chapter 9.

Petitions peaked in 2015 and have fallen since

The number of Form I-526 petitions tells a clear story. Filings rose sharply through the mid-2010s, peaking at 17,691 in 2015. After that they declined, falling to 1,147 in 2022 and 2,792 in 2023. The guide attributes the post-2015 slide mainly to slower-than-expected processing and a growing backlog of applicants waiting their turn.

Who actually receives the visas

Visa issuance shows a shift in nationality over time. In FY2014, China accounted for 9,128 of the 10,723 visas issued, with India at just 96. By FY2023, China’s count had dropped to 6,262, while India had risen to 815 after reaching 1,381 in FY2022. Vietnam and South Korea also grew over the same period. The visible dip in FY2020 and FY2021 reflects the COVID-19 slowdown.

These figures include principal investors, spouses and children. They also explain why country of origin matters. The guide notes that visa availability depends on the investor’s, or spouse’s, country of chargeability, which is usually the country of birth. Applicants from high-demand countries can wait longer, even when their paperwork is complete.

The agencies behind every step

Three bodies handle the process, and it helps to know who does what:

• USCIS is part of the Department of Homeland Security. Its Investor Program Office adjudicates the initial petitions, Form I-526 or I-526E, and sends approved cases to the National Visa Center where required. USCIS also handles applications for adjustment of status (Form I-485) and removal of conditions (Form I-829).
• The National Visa Center (NVC) sits within the State Department. It receives approved petitions for investors living abroad, gathers personal documents, and forwards files to the relevant embassy or consulate.
• U.S. embassies and consulates conduct the visa interviews and issue the visas.

— — Xumit Capital team (xumitcapital.com)

Written by Margret Syriac (margret.tapmimpl2022@learner.manipal.edu)

Corporate governance holds immense significance for businesses and society as a whole. The recognition of massive corporate collapses resulting from weak governance systems has underscored the need for international improvements and reforms in corporate governance. Instances such as Enron and Parmalat have prompted countries worldwide to take pre-emptive measures to address similar issues within their domestic frameworks.

The USA responded swiftly to these failures by enacting the Sarbanes-Oxley Act in July 2002. Similarly, the UK published the Higgs Report and the Smith Report in January 2003 in direct response to recent corporate governance failures. These events have elevated corporate governance to become one of the most frequently discussed topics in the global business arena.

The importance of a strong and functional corporate governance can be further explained using instances where its absence led to collapse of largest companies on the face of Earth.

The high-profile collapse of Enron in 2001, one of America’s largest companies, has drawn international attention to corporate failures and highlighted the role that strong governance plays in preventing such incidents. Corporate governance encompasses both formal and informal institutions, including private and public entities, that govern the relationship between corporate insiders (those managing corporations) and all other stakeholders who invest resources in various companies.

Essentially, corporate governance comprises a set of rule-based processes, laws, policies, and accountability mechanisms that regulate the interactions between investors (shareholders) and the management of investee companies. The significance of corporate governance gained prominence in the aftermath of the Asian financial crisis in 1997–1998 and the corporate scandals of the early 2000s, such as Enron and WorldCom.

The Enron case highlighted several corporate governance problems. Unrestrained power in the hands of the CEO was a key issue, as well as unethical activities that continued to be revealed after the company’s downfall. For example, Enron’s energy traders manipulated markets where California purchased electricity, which became evident through released documents. Overall, Enron’s corporate governance was weak in various aspects, with a board lacking moral character and engaging in fraudulent behaviour.

The collapse of Enron led to significant reactions in the USA and the UK, prompting a focus on corporate governance weaknesses. The long-term impact of Enron’s collapse should ideally result in a cleaner and more ethical corporate environment worldwide. Regular updates to corporate governance codes and systematic reviews of checks and balances are essential to prevent similar incidents in the future.

Corporate governance checks and balances can help detect unethical practices, but they cannot cure them entirely. The intangible nature of fraud and the subjective nature of human behaviour create complexities in addressing issues of fraud and ethical breakdown. Sheldon Zenner, an American white-collar criminal and civil lawyer, emphasised the challenges in defining right and wrong during the Enron trial.

Unfortunately, after the threat of global financial crises waned, corporate governance took a backseat in academic research. However, the enduring lessons from these incidents serve as a reminder of the ongoing importance of robust corporate governance practices to ensure the stability and integrity of businesses and financial systems.

Let’s dive into events where corporate governance changed the future of the companies for good.

In the early 2000s, Nestlé, a renowned multinational food and beverage company, found itself entangled in a controversy surrounding unethical marketing practices and a perceived lack of transparency. These allegations had the potential to significantly impact the company’s reputation and, subsequently, its stock performance. However, Nestlé swiftly responded to the situation by implementing robust corporate governance measures, ultimately turning the tide in its favour.

Recognising the importance of addressing the concerns raised, Nestlé took proactive steps to improve its corporate governance framework. One notable action was the adoption of the Nestlé Corporate Business Principles, a comprehensive set of guidelines outlining the company’s commitment to ethical behaviour, respect for human rights, and environmental sustainability. By establishing clear principles and values, Nestlé demonstrated its dedication to responsible corporate conduct.

Furthermore, Nestlé recognised the significance of transparency in rebuilding trust with its stakeholders. The company strengthened its sustainability reporting practices, providing comprehensive and detailed information about its environmental and social performance. This commitment to transparency allowed investors and the public to gain a deeper understanding of Nestlé’s efforts in areas such as water management, climate change mitigation, and responsible sourcing of raw materials.

The implementation of stronger corporate governance measures had a positive impact on Nestlé’s stock performance. Investors appreciated the company’s proactive approach in addressing the allegations and its commitment to ethical conduct and sustainability. The enhanced transparency reassured stakeholders that Nestlé was taking the necessary steps to align its practices with societal expectations.

Transparency of corporate practices plays a significant role in influencing the stock prices of a company. Transparent companies tend to attract more investors, enjoy higher market confidence, and benefit from a positive reputation. On the other hand, a lack of transparency can erode investor trust, increase perceived risk, and negatively impact stock prices. Investors value transparency as it enables them to make informed decisions and assess the long-term prospects and sustainability of a company.

Facing scrutiny over its weak governance practices in the late 2000s, Samsung undertook significant reforms. These included appointing independent directors, enhancing disclosure practices, and strengthening internal controls. These efforts contributed to increased investor trust and positively impacted Samsung’s stock performance.

In India, the structure responsible for overseeing corporate governance initiatives consists of two key entities: the Ministry of Corporate Affairs (MCA) and the Securities and Exchange Board of India (SEBI). The role of SEBI is to monitor and regulate the corporate governance practices of companies listed on stock exchanges in India. This regulatory oversight is carried out through the implementation of Clause 49, which is incorporated in the listing agreement between stock exchanges and companies. Compliance with the provisions of Clause 49 is mandatory for listed companies.

On the other hand, the Ministry of Corporate Affairs, in collaboration with various committees and forums, facilitates the exchange of knowledge and ideas related to corporate governance. One such forum is the National Foundation for Corporate Governance (NFCG), which operates as a non-profit trust. The NFCG serves as a platform for corporate leaders, policymakers, regulators, law enforcement agencies, and non-governmental organisations to share their experiences and perspectives on corporate governance.

The Desirable Corporate Governance Code introduced by the Confederation of Indian Industry (CII) in 1998 brought forth the concept of independent directors and their compensation in listed companies. The Kumar Mangalam Birla Committee, in 2000, recommended that if a company had an executive Chairman, at least half of the board should consist of independent directors, while in other cases, at least one-third should be independent directors.

The Clause 49, which was updated based on the report by the Narayana Murthy Committee, provides a more detailed definition of independent directors. It also mandates listed companies to have an optimal mix of executive and non-executive directors, with non-executive directors accounting for at least 50% of the board.

In 2013, the Companies Act introduced the requirement of appointing a resident director and a woman director. The Act also defined the term “Key Managerial Personnel” to include positions such as Chief Executive Officer, Managing Director, Company Secretary, and Chief Financial Officer, among others. The Act introduced new concepts like performance evaluation of the board, committees, and individual directors.

The revised Clause 49, implemented in 2013, states that the compensation paid to non-executive directors, including independent directors, must be determined by the Board and approved by shareholders in a General Meeting. There are also limits placed on stock options granted to non-executive directors, and these details must be disclosed in the company’s annual report. Independent directors are required to adhere to a Code of Conduct and affirm their compliance with it annually.

Countries like India, experiencing rapid growth, have attracted significant investments from international investors and large domestic financial institutions seeking global expansion. As a result, corporate governance standards in investee companies have made notable advancements. Numerous research reports have demonstrated that companies with robust governance systems have delivered favourable risk-adjusted returns to their shareholders. Therefore, if a company wishes to attract institutional investor participation, it must demonstrate a compelling commitment to enhancing the quality of its corporate governance practices.

Indian companies should adopt best practices, such as the OECD Corporate Governance Principles revised in 2004, which serve as a global benchmark. Given the prevalence of concentrated corporate ownership in countries like India, it becomes crucial to treat all shareholders, including domestic and foreign institutional investors, fairly and equitably.

Institutional investors are expected to actively engage in the voting process during Annual General Meetings (AGMs) for the shares they hold in their portfolio companies. Additionally, they should publicly disclose their voting records, including the reasons behind any non-disclosures. It is also essential for institutional investors to provide explanations for their support or opposition to any Board Resolution of their portfolio companies, which should be made available on their websites.

Under the 2013 Act, it is now compulsory for companies with over one thousand shareholders, debenture-holders, deposit-holders, and other security holders at any point in a financial year to establish a Stakeholders Relationship Committee. This committee will be composed of a non-executive director serving as the chairperson, along with other members as determined by the Board. The primary objective of this committee is to address and resolve any concerns or complaints raised by the company’s security holders.

Corporate governance is a critical driver of stock market trading. By fostering transparency, accountability, and ethical behaviour, it builds investor confidence, protects shareholder rights, and mitigates risks. Moreover, effective governance practices contribute to long-term value creation and sustainable growth. As investors, it is imperative to assess a company’s governance structure before making investment decisions. Likewise, companies must prioritise corporate governance to maintain a competitive edge and attract investment.

Remember, a well-governed company not only benefits its shareholders but also contributes to the overall stability and integrity of the stock market.

Written by Rohin Maliakkal (rohin.tapmimpl2022@learner.manipal.edu)

Position sizing is the process of determining the appropriate size of each investment in a portfolio based on the investor’s risk tolerance, investment objectives, and the characteristics of the stock or asset under consideration. Position sizing allows investors to limit the exposure to a single stock or market sector and help in reducing risk.

Why is position sizing important?

Given below is an example of five different trades that happened in five different days. In the first example, without proper position sizing we could see a winning percentage of 60% but still it is a loss we end up losing our capital. Many of us may have experienced this. But look at the second example same winning percentage but different result. How did this happen?

There comes the reason why it is important to keep position size and the importance of risk to reward ratio. In the second case, we gave a certain percent of the capital to each trade thereby limiting exposure of the whole capital. And also, in each trade in a day, we decided a certain risk to reward ratio also. This helps in limiting the loss that can happen in a trade. Further moving on I’ll explain how can this be done.

There are basically 3 types of position sizing techniques:

1. Unit per fixed amount

This strategy is used by most people. In this strategy, a trader or investor decides how much money they are willing to risk on each trade or investment.

The trader or investor then decides on the unit size for each trade or investment. The unit size is the amount of money assigned to each trade or investment. For example, if we have a capital of 4 lakh and have an opportunity of 4 trades, allocate 1 lakh to each of the trades as shown in the figure below. As a trader I will allocate the number of units based on this fixed amount (1 lakh) which I am willing to risk with this approach. The price was taken on 9th May 2023.

From this method, if we look at the maximum loss that may come from a particular trade in this portfolio is ₨ 447 (after brokerage). This helps in limiting the losses and saving the capital. By this method, for starters we can set a risk to reward ratio of 0.3:0.6 % by setting a target and stop loss for each trade.

2. Percentage margin

Another method used by traders and investors to determine the appropriate size of their positions is percentage margin in position sizing. Each trade or investment is assigned a percentage of the trading account.

To use this method, we must first decide how much money we are willing to risk on each trade as a percentage of their account size. Assuming we have a capital of 1 lakh, we decided to assign 5% of our capital to a trade that is 5000 rupees. Suppose we get an opportunity to trade HINDUNILVR (now trading at about 2500), we can buy 2 shares of HINDUNILVR as per our 5% capital. If we also get an opportunity to buy ULTRACEMCO (now trading at Rs.7735), we should let go off the opportunity as this is not included in our method.

It is important to note that using percentage margin for position sizing can result in larger position sizes as the trading account grows, increasing the risk of significant losses if a trade goes against the trader or investor. As a result, as the trading account grows, it is critical to constantly reassess and adjust the percentage margin.

3. Percentage volatility method

The volatility of the underlying asset is used to determine the appropriate position size for a trade. Suppose TCS on 4th May 2023 has O:3,180 H: 3,224 L: 3,178 C: 3,220, so the volatility is found by subtracting high and low price of that day.

Volatility = 3,224–3,178 = 46

We can find the volatility of TCS for the last n days and average it with the help of Average True Range (ATR) indicator. Here for example ATR of TCS for the last 14 days is around 50.

So now we have to find how many shares we have to buy assuming a capital of 10 lakh and if we decide on a volatility exposure of 0.5%. i.e., about 5K. So, I should buy shares such that the overall volatility exposure is less than 5K.

5000/50 = 100 shares. That means 100*3,220 = Rs. 3,22,000

So, if the position goes against us, we stand a chance to lose 5k on 10 lakh capital. This same can be done in a portfolio. Let’s look into a portfolio in which it’s done. From the same example given in units per fixed amount. Let the stocks be TATA STEEL, ONGC, HINDUNILVR, JSW STEEL. So a volatility of 1% is determined for 10 lakh capital which is distributed uniformly among the 4 stocks.

That is Rs. 2,500/- each

From a capital of 10 lakh, only Rs. 5,57,130 has to be employed in the portfolio. The remaining amount can be used to invest in other trade opportunities. Even if all positions of this portfolio go down also, we stand to lose 10k. The volatility percent is decided by a trader’s risk tolerance towards his investment. By these methods I hope for a trader or an investor you could book high profits by minimizing your losses.

Written by Margret Syriac (margret.tapmimpl2022@learner.manipal.edu)

Introduction

The financial industry has greatly benefited from technological advancements over the past 20 years. Most capital markets professionals believe that technology has improved market efficiency, although some industry veterans have lost their jobs due to automation. Baby boomers were less positive about the impact of technology than Gen Xers and millennials, and regulators are still grappling with the occasional volatility caused by computer-based trading. However, the advantages of technological progress far outweigh the disadvantages, with technology helping both big companies and small startups to thrive in the market. Although more than half of the respondents believe that technology has helped established companies to solidify their position, a larger majority believe that it has also enabled new startups to challenge the market leaders, despite the inherent difficulties of breaking into an entrenched market.

Capital markets professionals overwhelmingly see artificial intelligence as the most potentially disruptive technology in the coming 3–5 years.

WHAT’S NEW?

Trading desks have become more efficient and have improved their execution quality with the use of algo wheels for equity trades and automated responses to bond market requests. Large data sets and complex analytics have also enabled asset managers to get the best price on trades and work with the best counterparties, ultimately saving investors millions. Banks are using machine learning to provide better service to clients based on their behavior and preferences. Although these advancements have had a significant impact, they only represent the beginning of what’s possible in the future.

1. Artificial Intelligence

The use of artificial intelligence (AI) in the stock market is rapidly transforming the industry. By leveraging AI algorithms, traders can analyse vast amounts of data, detect patterns, and make better-informed decisions about purchasing and selling stocks. These algorithms are capable of learning from past decisions, allowing them to refine their strategies and become more effective over time. Additionally, AI can assist traders in monitoring market conditions and news updates in real-time, offering valuable insights into market sentiment and potential risks.

2. Blockchain

The stock market is experiencing a transformation through the use of blockchain technology. Blockchain leverages a distributed ledger that is decentralized, which can enhance transparency and efficiency in the stock market. By using blockchain-based systems, transactions can be processed with greater speed and security, reducing the likelihood of fraud and errors. Furthermore, intermediaries may become redundant because of blockchain technology.

3. Robotic Process Automation

RPA is a technology that automates repetitive tasks and processes using software robots. In the stock market, RPA can be used to automate tasks such as data entry, data analysis, and trade execution. By freeing up traders from these tasks, RPA can help them focus on higher-level tasks such as strategy development and risk management.

4. Quantum computing

Quantum Computing is a new technology that has the potential to revolutionize the working of stock market. With its ability to process large amounts of data and perform complex calculations at unprecedented speeds, quantum computing can help traders analyze market data more quickly and accurately. It can also help develop more sophisticated trading strategies that take into account a wider range of variables and factors.

5. Internet of Things (IoT)

IoT is a network of interconnected devices and sensors that collect and transmit data over the internet. In the stock market, IoT can be used to collect data from a variety of sources such as social media, news articles, and market data. This data can then be used to develop trading strategies that consider a wide range of variables and factors.

TECHNOLOGY THAT MATTER

Performance analytics sit at the intersection of several bigger technology trends, including AI, cloud computing, alternative data, and big data.

The current state of the market should be considered a new standard, and the industry will aim to exceed it each year. Automated trading, alternative data, and real-time analytics are now the norm, and the focus should shift to emerging technology tools that will have a greater impact in the next three to five years.

Trading desks need to consider both tactical and strategic innovation. Although they have no effect on the desk’s P&L today or next month, the former, which includes revolutionary technological advancements like artificial intelligence and cloud computing, is essential for long-term growth and profitability. Solutions focused on cybersecurity, real-time risk management and trade performance analytics will all be critical to the success of trading desks in the coming years.

TRADING AND AI

According to capital markets professionals, among the many incredible technologies that have impacted our lives, such as cloud computing, high-speed internet, and blockchain, AI is seen as the most potentially disruptive in the next three to five years. In contrast, institutional traders of all ages remain mostly uninterested in cryptocurrencies despite efforts to make these markets more accessible. Millennials are particularly optimistic about the potential for AI on trading desks, with two-thirds believing it will be highly disruptive in the coming years, compared to only half of Gen Xers. Having grown up with AI affecting various aspects of their lives, such as Netflix and Google, millennials are confident about the impact it can have on capital markets.

AI TRADING:

AI trading is the use of artificial intelligence algorithms to analyze and trade in the stock market. These algorithms use machine learning techniques to analyze vast amounts of market data, such as stock prices, news articles, and social media trends, to identify patterns and make informed trading decisions. AI algorithms can also learn from past decisions and adjust their strategies, accordingly, making them more effective over time.

How does it work?

AI trading works by using machine learning algorithms to analyze market data and make trading decisions. These algorithms use a variety of techniques, including natural language processing, sentiment analysis, and technical analysis, to identify patterns and trends in the market. Once a pattern or trend is identified, the algorithm can execute trades automatically based on predefined trading rules.

AI algorithms can learn from past decisions and adjust their strategies, accordingly, making them more effective over time. Finally, AI trading can help reduce the emotional biases that can often impact human traders, leading to more consistent and profitable trading strategies.

Some AI tools to aid stock trading:

• Sigmoidal: The program utilizes artificial intelligence to acquire knowledge on forecasting market trends and recognize connections among assets. Furthermore, machine learning automates the trading of securities. The firm’s artificial intelligence and robotic process automation services handle all aspects of the development process, from brainstorming to the finished product. This technology simplifies the examination of potential investments to help with decision-making.
• Kavout is an AI-powered stock selection and analysis platform that uses machine learning algorithms to identify promising investment opportunities. The platform uses a proprietary scoring system to rate stocks based on a range of factors such as financial performance, growth potential, and market sentiment.
• Trade Ideas is an AI-powered trading platform that uses machine learning algorithms to identify profitable trading opportunities. The platform scans the market for patterns and trends and provides traders with real-time alerts and recommendations based on their trading strategies.
• Tickeron: One of the many useful features offered by Tickeron is AI Trend Prediction. Tickeron utilizes an AI-driven trend forecasting system that examines historical pricing data to anticipate future market trends. Additionally, it assigns a confidence score to each prediction to enable you to evaluate its probability.

What’s the Impact?

The impact of artificial intelligence (AI) on financial services is still up for debate, but machine learning (ML) is seen as the most attention-grabbing type of AI. Despite challenges related to data normalization and privacy rules, capital markets firms are hiring specialists to effectively aggregate and normalize data for AI-driven solutions. Banks, exchanges, and fintech vendors are making the most progress with AI, with AI investment expected to be most impactful for execution algorithms and the analysis of structured and unstructured data. AI-driven execution algorithms have been used by a small number of early adopters in equity markets, and as they learn from market conditions and traders’ preferences, they can provide better services at a lower cost. Asset managers and hedge funds are not investing heavily in AI, with only 1 in 5 using it today. Given the challenges and costs associated with building cutting-edge AI solutions, it may be smarter for these firms to let their vendors and trading counterparties make the investment. This way, they can access fully vetted tools and reduce risk while still benefiting from AI on the trading desk.

Development of AI?

While financial services have invested billions of dollars in developing technology, the lack of sophistication in AI on trading desks is somewhat disappointing given the level of talent in the industry. However, the true potential of AI in trading has not yet been fully explored. Developing AI is expensive as it requires significant investment in R&D and developer resources. Nonetheless, the benefits of AI are expected to outweigh the costs, with costs reducing after the initial implementation. This is particularly important given the current pressures on broker-dealers to increase return on equity for their low-margin trading businesses. The buy side is also facing pressure to reduce costs due to the continued downward pressure on management fees.

Reference:

Survey results by Refinitiv examining the technology trends, the data explosion and the skills required to be successful in capital markets in the future : The future of Trading Technology

A Message from InsiderFinance

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the InsiderFinance Wire
• 📚 Take our FREE Masterclass
• 📈 Discover Powerful Trading Tools

What’s on the Way to your Trading Desk?: Technology That Matters was originally published in InsiderFinance Wire on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Kaushik Sanghani (kaushik.sanghani@xumitcapital.com)

Backtesting strategies using spot market and futures market data has become easier with the availability of various libraries like backtrader, vectorbt, and backtesting.py. However, backtesting trading strategies on derivatives is not as simple. While equity data is freely available on Yahoo Finance, most derivatives data is paid, especially for intraday data. Nsepy and Nse Bhavcopy provide free EOD option data for weekly and monthly expiry, but accessing and managing this data can be complicated due to different strike prices, expiries, and holidays. This article offers a Python code that can download daily Bank Nifty options data from Nsepy and organise it in a structured way for further backtesting. While the article doesn’t delve too deeply into the logic behind the code, it includes comments where necessary for clarity.

The process used to develop this code involved several steps.

1. Initially, the weekly expiry dates from 2017 to 2023 were identified.
2. These dates contain the expiry dates for all instruments, with BANK NIFTY and NIFTY expiring on Thursdays and FIN NIFTY expiring on Tuesdays, assuming no holidays. If the goal is to obtain weekly option data, then the expiry days that fall on Tuesdays should be excluded, and Mondays and Fridays should be removed after accounting for holidays.
3. The expiry date for the last week is determined, which corresponds to the FUTURE contract expiry date for that month.
4. The futures data for each expiry date is downloaded and saved to a dataframe.
5. Using nsepy, options data is downloaded for the given expiry and 10 strikes above and below the futures expiry prices are selected.
6. Finally, the data is organized, and the NIFTY cash price is added.

#importing libraries
import pandas as pd
from datetime import date
import nsepy
from copy import deepcopy
from datetime import datetime
import numpy as np
from nsepy import get_history

#-----------------------------------------------------------------------------------------------------

#creating dictionaries of expiry dates and then also to store weekday of that expiry to remove the 
#tuesday expiry of finnifty
expiry_dict = {2017:{},2018:{},2019:{},2020:{},2021:{},2022:{}}
expiry_dict_weekday = {2017:{},2018:{},2019:{},2020:{},2021:{}, 2022:{}}

month_dict = {}
month_dict_for_weekday = {}

#adding 1-12 month in month_expiry as well as month_dict_for_weekday
for i in range(1,13):
    month_dict[i] = []
    month_dict_for_weekday[i] = []

#getting expiry dates from nsepy.get_expiry_date for each month and year and then storing into our
#dictionary
#also calculating weekday of that expiry
#if it's monday=0, thursday=3 etc.
for year in expiry_dict:
    for month in month_dict:
        expiry_dict[year][month] = list(nsepy.get_expiry_date(year = year, month = month))
        expiry_dict[year][month].sort()
        
        expiry_dict_weekday[year][month] = []
        for k in expiry_dict[year][month]:
            expiry_dict_weekday[year][month].append(k.weekday())



#adding values for 2023 because we are in the march right now and we are not having data for all 12 months
expiry_dict[2023] = {1:[],2:[]}
expiry_dict_weekday[2023] = {1:[],2:[]}
for month in range(1,3):
    expiry_dict[2023][month] = list(nsepy.get_expiry_date(year = 2023, month = month))
    expiry_dict[2023][month].sort()
    
    expiry_dict_weekday[2023][month] = []
    for k in expiry_dict[2023][month]:
        expiry_dict_weekday[2023][month].append(k.weekday())
        
#-----------------------------------------------------------------------------------------------------



#creating copy
expiry_dict_after_removing_fin_exp = deepcopy(expiry_dict)
expiry_dict_after_removing_fin_exp_weekday = deepcopy(expiry_dict_weekday)

#removing the monday, tuesday and friday expiry
for year in expiry_dict:
    for month in expiry_dict[year]:
        for day in range(len(expiry_dict_weekday[year][month])):
            if ((expiry_dict_weekday[year][month][day]) == 1 or (expiry_dict_weekday[year][month][day]) == 4 or (expiry_dict_weekday[year][month][day]) == 0): #and (year == 2021 or year == 2022):
                
                print(expiry_dict_weekday[year][month][day])
                expiry_dict_after_removing_fin_exp[year][month].remove(expiry_dict[year][month][day])
                expiry_dict_after_removing_fin_exp_weekday[year][month].remove(expiry_dict_weekday[year][month][day])
#-----------------------------------------------------------------------------------------------------



#getting last month expiry for future contract
last_exp_of_month = []


for year in expiry_dict_after_removing_fin_exp:
    for month in expiry_dict_after_removing_fin_exp[year]:
        last_exp_of_month.append(expiry_dict_after_removing_fin_exp[year][month][-1])
#-----------------------------------------------------------------------------------------------------


#creating sample format for banknifty futures data
sample_data = get_history(symbol = 'BANKNIFTY', index= True, futures= True, start =date(2019, 2, 28) , end = date(2019, 3, 28), expiry_date=date(2019, 3, 28))
sample_data.columns

sample_data

#column names of banknifty futures data
bank_nifty_fut_data = pd.DataFrame(columns=sample_data.columns)
bank_nifty_fut_data

#-----------------------------------------------------------------------------------------------------


#getting bank nifty futures data for each month expiry and appending future data into one dataframe
for i in range(len(last_exp_of_month)-1):
    df = get_history(symbol = 'BANKNIFTY', index=True, futures=True, start=last_exp_of_month[i] , end = last_exp_of_month[i+1], expiry_date=last_exp_of_month[i+1])
    
    bank_nifty_fut_data = pd.concat([bank_nifty_fut_data,df])

#-----------------------------------------------------------------------------------------------------



#all option expiries in the list format
all_option_expiries = []

for year in expiry_dict_after_removing_fin_exp:
    for month in expiry_dict_after_removing_fin_exp[year]:
        for exp_date in expiry_dict_after_removing_fin_exp[year][month]:
            all_option_expiries.append(exp_date)
all_option_expiries

#we want to work with weekly expiries and for bank nifty it has started from second month so removing
#expiry data of january 2017
all_option_expiries = all_option_expiries[4:]
all_option_expiries

#creating sample dataframe for the option data
sample_api_call = get_history(symbol = 'BANKNIFTY', index=True,start =date(2019, 1, 31),end =date(2019, 2, 14),option_type='PE',strike_price=10000,expiry_date=date(2019, 2, 14))

options_data = pd.DataFrame(columns=sample_api_call.columns)
options_data

#-----------------------------------------------------------------------------------------------------

#iterating over all the expiries
for n in range(len(all_option_expiries) - 1):
    print(all_option_expiries[n])
    #getting future price at current expiry
    bank_nifty_fut_price_at_given_date = bank_nifty_fut_data['Close'].loc[all_option_expiries[n]]
    
    if (isinstance(bank_nifty_fut_price_at_given_date, pd.Series)):
        bank_nifty_fut_price_at_given_date = bank_nifty_fut_price_at_given_date[0]
        
    #we will be fatching data of +10 strike and -10 strike from the expiry price of bank nifty 
    #future
    bank_nifty_fut_price_at_given_date_rounded = round(bank_nifty_fut_price_at_given_date/100)*100
    
    strike_low_range = bank_nifty_fut_price_at_given_date_rounded - 1000
    strike_upper_range = bank_nifty_fut_price_at_given_date_rounded + 1000
    #below 100 is a strike price difference. for bank nifty it's 100 now and for nifty it's 50 now
    for i in range(strike_low_range,strike_upper_range,100):
        
#         print(i)
        #getting pe and ce data for the given expiry, strike price and given underlying
        df_pe= get_history(symbol = 'BANKNIFTY',index = True,start = all_option_expiries[n], end = all_option_expiries[n+1],option_type='PE',strike_price=i,expiry_date=all_option_expiries[n+1])
        df_ce = get_history(symbol = 'BANKNIFTY',index = True,start = all_option_expiries[n], end = all_option_expiries[n+1],option_type='CE',strike_price=i,expiry_date=all_option_expiries[n+1])
        #storing it into options_data
        options_data = pd.concat([options_data,df_pe])
        options_data = pd.concat([options_data,df_ce])
        
#-----------------------------------------------------------------------------------------------------



#future data
bank_nifty_fut_data = bank_nifty_fut_data[['Close']]

#taking necessary columns from option data
options_data = options_data[['Symbol','Expiry','Option Type', 'Strike Price', 'Close']]
options_data

#adding values bank nifty cash and bank nifty future
options_data['BANKNIFTY FUT'] = 0
options_data['BANKNIFTY CASH'] = 0


#getting bank nifty cash data for the given interval
bank_nifty_cash = get_history('BANKNIFTY',index=True, start = date(2017,2,2), end = date(2023,2,28))
bank_nifty_cash = bank_nifty_cash[['Close']]


#attaching spot price of bank nifty for given each row of option price on given table
for i in range(len(options_data)):
    print(options_data.index[i])
    options_data['BANKNIFTY CASH'].iloc[i] = bank_nifty_cash['Close'].loc[options_data.index[i]]
    #options_data['NIFTY FUT'].iloc[i] = nifty_cash_data['Close'].loc[options_data.index[i]]
    
#-----------------------------------------------------------------------------------------------------


options_data_with_reset_index = options_data.reset_index()
options_data_with_reset_index

#creating the price pivot table for call and put 
opttable = pd.pivot_table(options_data_with_reset_index, values='Close', index=['index','Option Type','Expiry','BANKNIFTY CASH'], columns=['Strike Price'], aggfunc=np.sum)


#grp based on option type
grouped_with_pe_ce = options_data.groupby('Option Type')
grouped_with_pe_ce_after_grouping = grouped_with_pe_ce.apply(lambda a: a[:])

#just pe data
options_data_pe = options_data[options_data['Option Type'] == 'PE']

#just ce data
options_data_ce = options_data[options_data['Option Type'] == 'CE']

#pe ce data in one data frame for given date
ce_pe_in_differnt_columns = pd.concat([options_data_ce,options_data_pe], axis = 1)

#-----------------------------------------------------------------------------------------------------

Notes:

The code provided above is intended for use in a notebook environment. To fully comprehend the logic and nuances of the code, it is recommended that you run it in a notebook environment. The code in different notebook cells is separated by “# — — — — -”.

If you wish to download stock option data, a similar approach can be used. Simply download the data for month-end expiries and apply the same logic to FIN NIFTY by excluding Thursday expiries and using Wednesday for holidays.

In the final cell, you can export the data in various formats, including one dataframe containing all CE PE data, a pivottable format, and separate dataframes for CE PE data, as well as a row containing both CE PE data and the NIFTY cash price.

Examples of the exported data are provided below:

The above downloaded data can be utilised for backtesting various option End of Day (EOD) trading strategies. We will investigate this further in the future. Thank you for taking the time to read this article.

A Message from InsiderFinance

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the InsiderFinance Wire
• 📚 Take our FREE Masterclass
• 📈 Discover Powerful Trading Tools

How to Download Bank Nifty Options Data using Python? was originally published in InsiderFinance Wire on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Kaushik Sanghani (kaushik.sanghani@xumitcapital.com)

In my previous articles, I primarily utilized the moving average crossover strategy to explain quantitative concepts, despite the fact that it is not a widely used trading strategy. Most traders rely on tested technical indicator strategies to make trading decisions. However, there is another option available — AI and Machine Learning. AI and ML have become ubiquitous in many industries. As beginners, we are familiar with typical machine learning projects such as house price prediction models and recommendation systems. In this article, we will explore how to use machine learning algorithms on stock market data from scratch in a straightforward manner. Additionally, we will perform back-testing on our trading strategy. While reading this article, readers will have the opportunity to implement the code in their Python environment.

It’s worth noting that this article does not provide a guaranteed strategy for making money in the live market. Instead, it teaches readers how to approach machine learning when developing and back-testing strategies. It provides a framework through which any machine learning algorithm can be back-tested on OHLC data.

To tackle this problem of using machine learning in NIFTY OHLC data, we will follow a typical data science framework which is described below.

1. Defining Problem Statement

2. Data Mining

3. Data Cleaning

4. Data Exploration

5. Feature Engineering

6. Predictive modelling

7. Evaluation of model

8. Visualization

9. Deployment

Now, let’s move on to the problem statement. The problem statement for any trading strategy is straightforward — “Should we buy an instrument at any given time?” If our algorithm predicts that the price of an instrument will rise in the future, we will buy it. On the other hand, if the algorithm predicts that the price will fall, we will take a short position on the instrument. In our strategy, we aim to predict whether we should buy NIFTY (assuming that we can replace it with NIFTYBEES) or not. This will be a long-only strategy.

This strategy uses supervised machine learning to evaluate the trading algorithm’s performance and back-test historical data. Therefore, we need to create a target variable. The target variable or signal will have two values — 1 and 0. A value of 1 means buying the instrument. In contrast, a value of 0 means not doing anything since it is a long-only strategy. We will create a target variable based on the near future return, one candle away.

#Importing libraries
import pandas as pd  
import numpy as np 
import quantstats as qs 
import talib as ta 
from copy import deepcopy
import seaborn as sns 
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay


import matplotlib.pyplot as plt 
import matplotlib.colors as clrs

from statsmodels.tsa.stattools import adfuller
from sklearn.model_selection import train_test_split

#Reading 1m file
df = pd.read_pickle('nifty_1m_cleaned_data.pickle')
df = df.reset_index()
df['index'] = df['index'] - pd.Timedelta(minutes=1)

# Assuming you have a dataframe named df with a DatetimeIndex
df = df.set_index('index')
df = df.resample('15min', offset='15min').agg({'Open': 'first', 'High': 'max', 'Low': 'min', 'Close': 'last'})
df = df.dropna()

df = df.tail(60000)

data = deepcopy(df) 

data['Open'] = np.float64(data['Open'])
data['High'] = np.float64(data['High'])
data['Low'] = np.float64(data['Low'])
data['Close'] = np.float64(data['Close'])
data.index = pd.to_datetime(data.index)

data.Close.plot(figsize = (8,5), color = 'b')
plt.ylabel('Close price')
plt.xlabel('Date')
plt.show()

#create a column 'future_returns' with the calculation of percentage change
data['future_returns'] = data['Close'].pct_change().shift(-1)

#create the signal column
data['signal'] = np.where(data['future_returns'] > 0, 1, 0)

To train our machine learning model, we require features that will aid in predicting future prices. As technical analysts, we know many indicators to help predict price trends and volatility. We can generate as many features as technical indicators. Our model will identify which combination of indicators and features works best with NIFTY data. We have used simple indicators such as previous candle returns, simple moving averages, correlation, volatility, and so on to create features to simplify matters. However, one can use other indicators to develop a better strategy. Here in the given code, the features and target variable are separated.

#Creating a column 'pct_change' with the 15-minute prior percentage change
data['pct_change'] = data['Close'].pct_change()

#Creating a column 'pct_change2' with the 30-minute prior percentage change
data['pct_change2'] = data['Close'].pct_change(2)

#Creating a column 'pct_change5' with 75-minute prior percentage change
data['pct_change5'] = data['Close'].pct_change(5)



#Creating a column by the name RSI, and assigning the RSI(14) value
data['rsi'] = ta.RSI(data['Close'].values, timeperiod=14)

#Creating a column by the name ADX, and assigning the ADX values to it
data['adx'] = ta.ADX(high = data['High'].values,
                    low = data['Low'].values,
                    close = data['Close'].values,
                    timeperiod = 14)

#Creating a column by the name sma and assigning SMA values to it
data['sma'] = data['Close'].rolling(window=14).mean()

#Creating a column by the name corr, and assigning the correlation values to it
data['corr'] = data['Close'].rolling(window=14).corr(data['sma']) 

# 1d 2d Volatility
data['volatility'] = data.rolling(int(6.25*4), min_periods = int(6.25*4))['pct_change'].std()*100 
data['volatility2'] = data.rolling(int(6.25*8), min_periods = int(6.25*8))['pct_change'].std()*100 

#Dropping missing values
data.dropna(inplace=True)

#Target
y = deepcopy(data[['signal']])

#Features
X = deepcopy(data[['Open','High','Low','Close', 'pct_change', 'pct_change2',
         'pct_change5', 'rsi', 'adx', 'sma', 'corr', 'volatility', 'volatility2']])

Let’s visualise all the features which we have created.

i = 1
#Set number of rows in subplot
nrows = int(X.shape[1])
for feature in X.columns:
    plt.subplot(nrows,3,i)
    
    #Plot the feature
    X[feature].plot(figsize=(12,3*X.shape[1]),
                   color = np.random.rand(3,))
    plt.ylabel(feature)
    plt.xticks(rotation='vertical')
    plt.title(feature)
    i += 1
plt.tight_layout()
plt.show()

Since most ML algorithms require stationary features, we will exclude non-stationary features. We will use the adfuller test to test for stationarity, which I have explained in the pairs trading article. After conducting the test, we have determined that Open, High, Low, Close, and SMA are not stationary features. Consequently, we will drop these features.

def stationary(series):
    result = adfuller(series)
    if(result[1] < 0.05):
        return 'stationary'
    else:
        return 'not stationary'
    
#Checking for stationarity
for col in X.columns:
    if stationary(data[col]) == 'not stationary':
        print(f'{col} -- not stationary')
        X.drop(columns=[col], axis=1, inplace=True)
    else:
        print(f'{col} -- stationary')

To enhance the model, we must eliminate multicollinearity from the feature space. Therefore, we will assess the correlation between all the features and exclude those with an absolute correlation greater than 0.75. One could set the threshold at 0.7. After conducting this test, we discovered that the volatility and volatility2 features correlate greater than 0.75. Hence, we will drop the volatility2 feature.

#plotting correlation
plt.figure(figsize=(8,5))
sns.heatmap(X.corr(), annot=  True, cmap = 'coolwarm')
plt.show()

#Function to return the paris with correlation above threshold
def get_pair_above_threshold(X,threshold):
    #Calculate the correlation
    correl = X.corr()
    
    #Unstack the matrix
    correl = correl.abs().unstack()

    #Recurring and Redudant pair
    pairs_to_drop = set()
    cols = X.corr().columns
    
    for i in range(0, X.corr().shape[1]):
        for j in range(0, i+1):
            pairs_to_drop.add((cols[i], cols[j]))

    #Dropping the recurring and redudant pair
    correl = correl.drop(labels=pairs_to_drop).sort_values(ascending=False)
    
    return correl[correl > threshold].index

print(get_pair_above_threshold(X,0.75))

X = X.drop(columns=['volatility2'],axis=1)

We will use the train_test_split method from the sklearn module to divide the data into 80% training and 20% testing data. Alternatively, one could choose a combination of (70, 30). Since we are working with time series data, we will not shuffle the data during the split. If we shuffle the data, we will encounter a problem of data leakage.

#Obtaining the features and target for the 'train_data' and 'test_data' without
#shuffling

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.80, shuffle=False)

plt.figure(figsize=(15,5))

plt.plot(X_train['pct_change'], linestyle = 'None',
        marker = '.', markersize = 3.0, label = 'X_train data',
        color = 'blue')
plt.plot(X_test['pct_change'], linestyle = 'None',
        marker = '.', markersize = 3.0, label = 'X_test data',
        color = 'green')

plt.title('Visulizing Train and Test Dataset')
plt.xlabel('Years')
plt.ylabel('% Change')
plt.legend()
plt.show()

Continuing from the previous paragraph, to create a machine learning model, we need to deal with a classification task as per the problem statement: whether to take a long position on NIFTY or not. We can use various machine learning algorithms, such as logistic regression, decision trees, SVM classifier, Random Forest, XG booster, etc. However, we will use the Random Forest Classifier for this article. This algorithm combines multiple decision trees to make more accurate predictions.

Each decision tree is trained on a random subset of the data and features, which makes the trees less prone to overfitting and increases the overall accuracy of the model. The algorithm then combines the predictions of each tree to make a final classification decision. Several hyperparameters, such as max_depth, max_features, and n_estimators, can be chosen randomly. However, we can use hyperparameter tuning methods to improve the algorithm further. Once we have chosen the hyperparameters, we can train our model, test it on the out-of-sample dataset and evaluate the results.

#Creating the machine learning model
rf_model = RandomForestClassifier(n_estimators=3, max_features=3,
                                 max_depth=2, random_state=4)

#Training the model
rf_model.fit(X_train, y_train['signal'])
"Training is done"

y_pred = rf_model.predict(X_test)
y_pred

accuracy_data = (temp == y_test['signal'])

accuracy_percentage = round(100* accuracy_data.sum()/len(accuracy_data),2)
accuracy_percentage

#plot accuracy data
plt.figure(figsize=(15,5))
#color mapping for the correct and incorrect predictions 
cmap = clrs.ListedColormap(['green', 'red'])
plt.yticks([])
plt.scatter(x = y_test.index, y = [1]*len(y_test),
           c = (accuracy_data!=True).astype(float),
           marker = '.', cmap = cmap)
plt.title('Accuracy (Correct is green and Incorrect is Red)')
plt.xlabel('Years')
plt.show()

#Define the confusion matrix
confusion_matrix_data = confusion_matrix(y_test.values,y_pred)

cm_display = ConfusionMatrixDisplay(confusion_matrix = confusion_matrix_data, display_labels = ['No Position', 'Long Position'])

cm_display.plot()
plt.show()

#Classification report 
classification_report_data = classification_report(y_test.values, y_pred)
print(classification_report_data)

The accuracy obtained for our algorithm is 51.39%, which is decent, as it predicts the correct results for 51.39 out of 100 times. However, relying solely on accuracy is insufficient to evaluate the model’s effectiveness. We also need to analyse the accuracy label-wise or the actions taken by the ML model. This involves assessing the total number of True Positive, True Negative, False Positive, and False Negative cases, which can be seen in the confusion matrix. Additionally, we have calculated the f1 score, which ranges from 0 to 1, with zero being the worst and one being the best. Explaining the confusion matrix and f1 score would require an entire article.

So far, we have obtained an accuracy of around 51%, which does not provide any information about the trading system. Therefore, we need to perform the typical back-testing process to learn more about this trading strategy and evaluate various metrics such as the equity curve of the strategy, returns, risk, Sharpe ratio, maximum drawdown, winning amount per trade, and losing amount per trade. We will calculate all these metrics for our strategy from the beginning. In the end, we will also use the quantstats library to compute more detailed metrics and visualize them.

#making strategy data for backtesting purpose
strategy_data = data[['Close']].merge(X_test[[]], left_index=True, right_index=True)

strategy_data['signal'] = y_pred
strategy_data



#Calculating the percentage change
strategy_data['pct_change'] = strategy_data['Close'].pct_change()

#Calculating the strategy returns
strategy_data['strategy_returns'] = strategy_data['signal'].shift(1) * strategy_data['pct_change']

#Dropping missing value
strategy_data.dropna(inplace=True)

strategy_data

#Trade logs

def get_trades(data, close_column, signal_column):
    #Create a dataframe to store value of each trade
    trades = pd.DataFrame()
    
    #Set current position as 0,
    #and entry time as empty
    current_position = 0
    entry_time = ''
    
    #Loop through each trade in dataframe
    
    for i in data.index:
        #Each row in dataframe will be treated as a new position
        
        new_position = data.loc[i, signal_column]
        
        if new_position != current_position:
            if entry_time != '':
                entry_price = data.loc[entry_time, close_column]
                exit_time = i 
                exit_price = data.loc[exit_time, close_column]
                
                trade_details = pd.DataFrame([
                    (current_position, entry_time, entry_price, exit_time, exit_price)
                ])
                trades = trades.append(trade_details, ignore_index=True)
                
                entry_time = ''
            
            if new_position != 0:
                entry_time = i 
                
            current_position = new_position
    
    #Adding the trade details to the 'trades' dataframe
    
    trades.columns = ['Position', 'Entry Time', 'Entry Price', 'Exit Time',
                     'Exit Price']
    
    trades['PnL'] = (trades['Exit Price'] - trades['Entry Price']) * trades['Position']
    
    return trades 

trade_log = get_trades(strategy_data, 'Close', 'signal')


trade_log

def get_analytics(trades):
    # Creating the dataframe to store the parameters
    analytics = pd.DataFrame(index=['Strategy'])
    
    # Number of long trades
    analytics['num_of_long'] = len(trades.loc[trades.Position == 1])
    
    # Number of short trades
    analytics['num_of_short'] = len(trades.loc[trades.Position == -1])
    
    # Total number of trades
    analytics['total_trades'] = analytics.num_of_long + analytics.num_of_short
    
    # Gross Profit
    analytics['gross_profit'] = trades.loc[trades.PnL > 0].PnL.sum()
    
    # Gross Loss
    analytics['gross_loss'] = trades.loc[trades.PnL < 0].PnL.sum()
    
    # Net Profit
    analytics['net_profit'] = trades.PnL.sum()
    
    # Profitable trades
    analytics['winners'] = len(trades.loc[trades.PnL > 0])
    
    # Loss-making trades
    analytics['losers'] = len(trades.loc[trades.PnL <= 0])
    
    # Win percentage
    analytics['win_percentage'] = 100 * analytics.winners /analytics.total_trades
    
    # Loss percentage
    analytics['loss_percentage'] = 100 * analytics.losers /analytics.total_trades
    
    # Per trade profit/loss of winning trades
    analytics['per_trade_PnL_winners'] = trades.loc[trades.PnL > 0].PnL.mean()
    
    # Per trade profit/loss of losing trades
    analytics['per_trade_PnL_losers'] = trades.loc[trades.PnL <= 0].PnL.mean()
    
    analytics = round(analytics, 2)
    
    return analytics.T

analytics = get_analytics(trade_log)


print(analytics)

#Plotting equity curve

strategy_data['cumulative_returns'] = (1 + strategy_data['strategy_returns']).cumprod()


#Plot cummulative strategy returns

strategy_data['cumulative_returns'].plot(figsize = (8,5), color = 'green')
plt.title('Equity Curve', fontsize = 14)
plt.ylabel('Cummulative returns')
plt.tight_layout()
plt.show()

cummulative_returns = (strategy_data['cumulative_returns'][-1] - 1) * 100
print(f'The cummulative returns is {cummulative_returns}')

#Annualised returns
annulised_return = ((strategy_data['cumulative_returns'][-1] ** (252*6.25*4/strategy_data.shape[0]) - 1))* 100
print(f'Annulised return is {annulised_return}')

#Annulised volatility
annulised_volatility = strategy_data['strategy_returns'].std()*np.sqrt(252*6.25*4)*100
print(f'Annulised volatility is {annulised_volatility}')

#Calculating running maximum

running_max = np.maximum.accumulate(strategy_data['cumulative_returns'].dropna())

#Ensure that the value nver drops below 1
running_max[running_max < 1] = 1 

#Calculate the percentage drawdown
drawdown = ((strategy_data['cumulative_returns'])/running_max - 1) * 100

#Calculate maximum drawdown
max_dd = drawdown.min()
print(f'Maximum drawdown is {max_dd}')

#Plotting under water equity curve

fig = plt.figure(figsize=(10,5))
plt.plot(drawdown, color = 'red')
plt.fill_between(drawdown.index, drawdown.values, color = 'red')
plt.title('Underwater equity curve')
plt.ylabel('Drawdown(%)')
plt.xlabel('Year')

plt.tight_layout()
plt.show()

#Sharpe ratio 
sharpe_ratio = round(strategy_data['strategy_returns'].mean() / strategy_data['strategy_returns'].std() * np.sqrt(252*6.25*4),2)
print(f'Shapre ratio is {sharpe_ratio}')

# Quantstats Report Making


strategy_data

strategy = {
    'strategy_returns': 1
}

strategy_data = strategy_data.reset_index()
strategy_data['index'] = pd.to_datetime(strategy_data['index'])
strategy_data = strategy_data.set_index(strategy_data['index'])

portfolio = qs.utils.make_index(strategy,rebalance=None,returns=strategy_data)


qs.reports.html(portfolio,title='Random Forest on NIFTY',download_filename='ml_in_market.html')

According to our analysis, our strategy generates approximately 11% return per year with a volatility of approximately 14% and a maximum drawdown of 15%, resulting in a Sharpe ratio of 0.83. However, we have not yet considered slippage and brokerage charges. Therefore, this strategy may not perform well enough to trade in the live market. But there are numerous possibilities to create different optimised trading strategies, such as using complex indicators, different machine learning algorithms with tuned hyperparameters, and risk management systems to reduce drawdown.

I hope this article has given you a clear understanding of the framework and approach to using ML in the stock market.

A Message from InsiderFinance

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the InsiderFinance Wire
• 📚 Take our FREE Masterclass
• 📈 Discover Powerful Trading Tools

Machine Learning (ML) Trading Strategy on NIFTY was originally published in InsiderFinance Wire on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Kaushik Sanghani (kaushik.sanghani@xumitcapital.com)

I am currently investigating high-frequency trading, which involves executing thousands of orders within seconds. This type of trading is quite distinct from technical trading and investment strategies. Fewer resources are available to learn about it than typical algorithmic trading strategies. To comprehend high-frequency trading, we must understand order books and order types. This article will provide a brief overview of how order books operate and the various types of orders. In future articles, we will also delve into specific strategies, such as statistical arbitrage and order book strategies.

Firstly, let’s begin by exploring how the order book operates. Most of us have likely come across the order book while placing orders. For example, I have attached a screenshot of Zerodha’s order book for Infosys.

The order book displays all the bids on the left-hand side and all the asks on the right-hand side. To illustrate how an order book is created, let’s use the following dummy data.

Let’s understand the order flow first. When a trader places an order, it is first directed to the broker, who then sends it to the central exchange, such as NSE or BSE. A matching algorithm is used at the exchange to match the buy order with the sell order, and a trade occurs. Most exchanges use the “Price Time Priority algorithm” to match orders based on their price and the time they were placed. To visualise these orders, the order book is maintained using the price time priority algorithm.

Price time priority algorithm:

According to this algorithm, the exchange will prioritise orders with the best price, and if two orders have the same price, the earlier order will be executed first.

Using the example above with dummy data, the generated order book is presented below.

In the current market, the price of a dummy asset is 995 rupees, and the order book is constantly updated as new orders are received and matched using the algorithm we discussed earlier. The changes in the order book depend on the type of orders placed by traders. Various types of orders, including limit orders and market orders, are well-known. However, to understand the market microstructure, we need to explore these order types further and discuss other order types.

To better understand different types of orders, it’s essential to understand the concept of a market maker. While many retail traders use directional strategies, some firms make profits by taking advantage of the spread, which is the difference between the best bid and best ask prices. In the example above, the spread is 5 rupees, which can vary depending on the security’s liquidity. Market makers profit by placing limit buy orders just below the best bid and limit sell orders just above the best ask, as in the case where the current market price is 990. By doing so, they can make money through the spread and are referred to as market makers.

Market Order:

A market order is an order type that is immediately executed at the best available price once it hits the exchange. If a trader places a market buy order, their order will be matched with the volume on the ask side, regardless of the price. These orders offer the advantage of execution certainty, but the trader pays the bid-ask spread, and when trading with a large volume, they may receive an inferior price. In the example given, with a current market price of 995 rupees, if a trader places a market buy order for 80 shares, their cost would be calculated as follows:

Cost = 1000*15 + 1005*50 + 1010*15 = 15000 + 50250 + 15150 = 80500

Limit Order:

In the context of the order book example given earlier, a limit order is an order type where the trader specifies the price and volume. If the specified price is better than the current market price, the order is immediately executed, otherwise, it is placed on the limit order book and will get executed when it matches with the opposite side trade. There is a fixed price when the order is executed, but there is no guarantee of getting filled, and the probability depends on the limit order price. For example, if a trader places a limit buy order at 994 when the current market price is 995, the order will be placed in the limit order book. If the trader places a limit buy order at the best ask (here 1000), it will get executed at that price. If the order is only partially filled, the remaining volume will be placed at price 1000 in the limit order book.

Stop order:

A stop-loss order is an instruction to sell (or buy) a security if it reaches a certain price level. It is a risk management tool traders, and investors use to limit their losses and protect their profits. Stop-loss orders can be set as a fixed price or percentage of the current market price. Once the stop-loss price is reached, the order becomes a market order and is executed at the next available price, which may be above or below the stop-loss level.

Iceberg order:

Iceberg orders refer to a type of order where a large volume of a trader is split into visible and hidden portions, much like how only a tiny part of an iceberg is visible above the water. The visible part is shown to other market participants, while the hidden part is kept private. This allows the trader to hide their true trading intentions and avoid revealing their entire trading volume at once. The hidden portion of the order is only released to the public domain after each execution of the order.

Fill or Kill order (FOK):

Fill or Kill (FOK) is an order where the trader instructs the broker to fill the entire order immediately or cancel. If the order cannot be filled completely, it gets cancelled, unlike other order types that can remain in the order book for some time. FOK orders are helpful for traders who need to execute a large order quickly or those who want to avoid partial fills. However, the disadvantage of FOK orders is that they may result in missed opportunities if the entire order cannot be filled.

In addition to the basic order types, there are more advanced orders, such as Gorilla orders, Pegged orders, and Discretionary orders. There are also other concepts in high-frequency trading, including dark pools. These topics will be covered in the following article.

A Message from InsiderFinance

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the InsiderFinance Wire
• 📚 Take our FREE Masterclass
• 📈 Discover Powerful Trading Tools

HFT Concepts: Order book and Order types was originally published in InsiderFinance Wire on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Shubham Soni (associate.01@xumitcapital.com)

Introduction

Mean reversion is a popular trading strategy used in the financial market to take advantage of the tendency of an asset’s price to move back to its mean or average price over time. This strategy uses statistical analysis and mathematical models to identify opportunities to buy low and sell high, thereby maximizing profits. Python, a powerful and versatile programming language, has become the go-to tool for data analysis and computational finance due to its simplicity, speed, and a rich library of modules and packages. In this article, we will explore how to implement a mean reversion trading strategy using Python, from data collection and analysis to backtesting and execution. Whether you’re a seasoned trader or just starting out, this guide will provide you with the knowledge and skills to build and refine your own mean reversion trading strategy in Python.

What is Mean Reversion

Mean reversion is a widely used trading strategy that is based on the assumption that the price of an asset will tend to move back towards its average price over time. This strategy involves buying an asset when its price is lower than its mean, and selling it when its price is higher. In this way, traders can take advantage of the market inefficiencies and generate profits. At its core, mean reversion traders look for strong price movements and then try to identify when those prices are likely to reverse. The way to apply this strategy in the market is to seek out extreme events and then bet that things will revert back to nearer the average.

Drawbacks of using Mean Reversion

Main problems that traders face when using mean reversion strategy:

1. If a stock is dropped for a major reason; such as a structural change or if there’s real game changing news, the chances of that stock snapping back quickly and giving you a profit is much smaller.
2. Momentum is the enemy of this strategy.
3. Strong price movements can go on for a very long time and a reversal may not occur for a long period
4. Instead of the price reversing and meeting the moving average again, often price just consolidates after a strong move and the moving catches up with the price

Designing the Strategy

In order to design our trading strategy, we will consider 200 Days moving average as the guiding principle. If we take Bharti Airtel stock (NSE:BHARTIARTL) as a reference and look at its price action over the last 1 year, we see that whenever the stock price deviates from its long-term average value, sooner or later it reverts back to the long term mean value.

The graph below shows the mean reversion action. Here the blue line represents daily price action and orange line represents the 200 days moving average

Most retail traders/investors in India do not usually short a stock simply because they do not understand the complexities associated with short selling, hence our main focus while designing this trading strategy will be on buying low and selling high.

There are numerous other ways in which mean reversion theory could be applied in trading, either independently or in conjunction with other trading strategies however since the scope of this article is limited to the introduction of Mean reversion as a trading strategy, we will keep our strategy simple.

Let’s understand key terminologies that we will be using in the code before we start making our trading system:

1. 52 week low: lowest price a stock has traded at, in the past 52 weeks.
2. 52 week high: The highest price a stock has traded at, in the past 52 weeks.
3. Diff: 52 week high — 52 week low
4. Distance: Closing Price — 200 MA value
5. 200 Days Moving Average: A 200-day Moving Average (MA) is simply the average closing price of a stock over the last 200 days. They are used as trend indicators that reflect momentum in the price of the asset.

Our system will generate a buy signal when all of the following conditions are satisfied:

1. (52 week low + diff * 0.2) < stock price < (52 week low + diff * 0.8)

When stocks trade closer to their 52w high/low price and their prices start going up or down closer to their extreme values, there is a possibility that some news is driving the change and the fluctuation in price is not random, hence we avoid any stock trading outside these ranges i.e. we only consider stocks whose current price action is inside of the purple lines.

2. Stock Price < 200 days MA

This gives the stock a chance to correct itself, and revert to its mean value, which is higher than its current price.

3. Distance is in the top 5 percentile

This will make sure that the deviation from mean value is significant and the stock price is expected to rise significantly.

Whereas, a sell signal is generated if:

1. Stoploss is hit: There is a possibility that after buying a stock, the prices of stock might fall, in order to take this into account, we set a stoploss (maximum loss we’re willing to take before selling).
2. Profit Cap is reached: If, as anticipated, the prices start to rise then we have to decide at what point we want to sell the asset to capture the profit before the prices fall again. (Helps us to keep greed levels in check).

Coding the Strategy

Data Sources

• We get the NIFTY 50 universe data (tickers) from nifty’s own website.

#NIFTY 50 Universe

nifty_50_ticker_url = 'https://www1.nseindia.com/content/indices/ind_nifty50list.csv'
nifty_50 = pd.read_csv(nifty_50_ticker_url)

• Now that we have the tickers list available, we get the historical data for the tickers from a python library called yfinance.

#Downloading historic data from yfinance
historic_price = yf.download((nifty_50.Symbol + '.NS').to_list(), progress=True)['Adj Close']

Data Preparation

• We have historical data of 50 stocks from NIFTY 50 available with us, using which we’ll calculate the moving average of all stocks.

#calculate 200 days moving average
moving_avg = historic_price.rolling(200).mean()

• Now we create a python dictionary to store the updated datasets such them we can access them easily.

#Creating a dictionary of data with Ticker as the Key and merged df as value
final_data = {}
for ticker in nifty_50.Symbol:
    ticker = ticker + '.NS'
    merged_df = pd.merge(historic_price[ticker], moving_avg[ticker], left_index=True, right_index=True)
    merged_df.columns = ['Adj Close', '200MA']
    merged_df['distance'] = merged_df['Adj Close'] - merged_df['200MA']#((merged_df['Adj Close'] - merged_df['200MA'])*100)/merged_df['200MA']
    data_dict = {
        'price_df': merged_df,
        '52w_data' : price_df.loc[ticker.split('.')[0]]
    }
    final_data[ticker] = data_dict

Main Algorithm

Here we code the main algorithm using Python.

This piece of code

• creates additional columns in the data frame to store buy and sell data
• Calculates max_price, and min_price of the past 52 weeks for each observation.
• Generates buy signal when all the conditions are satisfied.
• Generates sell signal when stop-loss is encountered.
• Generates sell signal when Max profit cap is reached.

cum_pnl = 0
txns = []


#condition 1
#Adj Close value between 20-60%
for ticker in final_data:
    df = final_data[ticker]['price_df']
    df = df.iloc[-1250:,:]
    df['52w_max'] = np.nan
    df['52w_min'] = np.nan
    df['diff'] = np.nan
    df['top_5'] = np.nan


    for i in range(len(df)):
        # slice the dataframe to get the last 250 rows
        sliced_df = df.iloc[max(0, i-250):i+1]
        # calculate the maximum and minimum price in the sliced dataframe
        max_price = sliced_df['Adj Close'].max()
        min_price = sliced_df['Adj Close'].min()
        diff = max_price - min_price
        # do something with the maximum and minimum price
        df.iloc[i,3] = max_price
        df.iloc[i,4] = min_price
        df.iloc[i,5] = diff
        df.iloc[i,6] = df.iloc[i,2] < np.percentile(sliced_df.distance, 5)


    state = 'out'  # state can be 'out', 'long' or 'short'
    profit = 0
    stop_loss = 0
   
    for index, row in df.iterrows():
    # check if the state is 'out' and the current price goes below 200MA
        if state == 'out' and ((row['Adj Close'] > (row['52w_min'] + 0.2* row['diff'])) & (row['Adj Close'] < (row['52w_min'] + 0.6*row['diff']))) & (row['200MA'] > row['Adj Close']) & (row['top_5']):
            state = 'long'
            buy_price = row['Adj Close']
            stop_loss = buy_price * 0.9
            print('Bought {} on {} at {}'.format(ticker, index, buy_price))
       
        # check if the state is 'long' and the current price goes below stop loss
        if state == 'long' and row['Adj Close'] < stop_loss:
            state = 'out'
            sell_price = row['Adj Close']
            profit = sell_price - buy_price
            cum_pnl += profit
            txns.append([index, ticker, profit])
            print('Sold {} on {} at {} for a profit of {}'.format(ticker, index, sell_price, profit))
       
        # check if the state is 'long' and the profit is more than 20%
        if state == 'long' and (row['Adj Close'] - buy_price) / buy_price > 0.2:
            state = 'out'
            sell_price = row['Adj Close']
            profit = sell_price - buy_price
            cum_pnl += profit
            txns.append([index, ticker, profit])
            print('Sold {} on {} at {} for a profit of {}'.format(ticker, index, sell_price, profit))
print('final profit/loss is rs: {}'.format(cum_pnl))

Backtesting

Now that we have designed and coded our strategy, we backtest it over the historical data. It is advised that you should test your strategy as back in time as possible because it exposes our strategy to various events which have happened in past decades, such as recessions, bear markets, bull markets, market crashes, etc. It makes your strategy more robust.

Since we have taken all the companies in present-day NIFTY 50 in our portfolio, it is not possible for us to back our data that deep therefore, we test our data on the last 5 years and take it as the benchmark to compare it with the past 1 year data.

It is important to note that how successful our strategy is also dependent upon the values of parameters that we set i.e. our risk-taking ability (stop-loss percentage) and how much we want the stock prices to rise before we sell them (profit target). Hence we take a combination of various stop-loss and profit-target values and backtest our data over those as well to find the most profitable combination.

Looking at the table above, we can see that the most profitable setup among all is where we take stop-loss to be 15% and profit-target to be 29%. It has generated a return of 18000 rs. in the time span of 5 years.

It should also be noted that our strategy has not been very successful because it is a very simple strategy However when combined with other strategies, it might work well.

Presented below is the Cumulative Profit and Loss graph of our most successful strategy.

When tested on 2022 data, our strategy performed poorly as it generated a return of rs 2128 which is not surprising considering what the state of the market was in the whole year 2022.

Conclusion

One thing that has been observed about mean reversion is that a good mean reversion trade requires things to stay the same. When a stock drops 10 or 20 percent, there is usually a reason, and you can usually find out what it is. If it’s dropped for a major reason, such as a structural change, or if there’s real game-changing news, the chances of that stock snapping back quickly and giving you a profit are much smaller. When this happens, you get momentum, and this is obviously the enemy of a mean reversion strategy. But other times, a stock can drop sharply for less obvious reasons. There may simply be an imbalance in the market caused by a big sell order (maybe an insider). Or the stock may drop due to an overreaction to a short-term event (such as a terrorist threat, election result, or oil spill).

These are often the most opportune moments for mean reversion trades. We get a big move, but really, not an awful lot has changed. So mean reversion strategies require things to stay the same. You don’t want structural, intrinsic change when making a mean reversion trade, but you do want overreactions and illogical price movements. If you can find ways to quantify that, you will be on your way to developing a sound mean reversion trading strategy.

A Message from InsiderFinance

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the InsiderFinance Wire
• 📚 Take our FREE Masterclass
• 📈 Discover Powerful Trading Tools

Mean Reversion Strategy using Python was originally published in InsiderFinance Wire on Medium, where people are continuing the conversation by highlighting and responding to this story.

Written by Dheer Ved (dheer.ved@xumitcapital.com)

UK Innovator Visa

In the last blog, we discussed an Active Investor Program and saw Canada Starta-Up Visa. So today, we are going to explore one more program of Investor Immigration Program under the Active Investor Program, the UK Innovator Visa.

The United Kingdom, also known as the UK, is a sovereign country in western Europe. It comprises four constituent countries: England, Scotland, Wales and Northern Ireland. The UK has a rich history and culture, with notable contributions to literature, art and science. The official language is English and the currency is the British Pound.

UK Innovator Visa is a type of visa available for those who wish to set up a business or invest in a business in the United Kingdom. The innovator visa is for people with experience who want to start their own business in the UK. The initial visa is granted for a period of 3 years which can be extended for more 3 years. The investor is applicable for permanent residency, Indefinite Leave to Remain (ILR) after 5 years of continuous residency and not being absent from the UK for more than 180 days in a 12-month period. Once the investor gets an ILR and holds it for 12 months, he/she can apply for citizenship.

What is the UK Innovator Visa?

On March 29 2019, the UK Home Office launched a new category known as “Innovator Visa” for experienced entrepreneurs seeking to establish a new business in the United Kingdom. This new category aims the businesses should be viable, scalable and innovative in order to drive economic growth in the UK. Your business or business idea must be approved by an Endorsing body in the UK and you meet other eligibility criteria.

Eligibility Requirements

In order to qualify for the Innovator Visa you must meet certain eligibility criteria such as

• Have a business idea that is Innovative, Scalable and Viable: Your business idea must be:

1. Innovative: The applicants must have a business plan that is new and different from existing market
2. Scalable: A structured business plan that the business will grow in the future and create necessary jobs
3. Viable: The business idea must be realistic and the applicant has relevant skills and experience to run the business

• The business or business idea must be approved by an Endorsing Body:

Applicant must obtain an endorsement letter from an approved endorsement body. Endorsing bodies are responsible for assessing the eligibility of the start-ups. These bodies can be business incubators, investment funds, universities, colleges and other bodies authorised by UK Home Office. The endorsement letter must be issued not more than 90 days before you apply for UK Innovator Visa.

• Investment Fund: You must have at least £50,000 in investment funds to apply for an innovator visa. You must prove where you got these funds and they must be legally sourced. You are allowed to make a joint investment with other Entrepreneurs but you cannot share the same investment. Each one of you is required to make a separate £50,000 investment. You do not need to invest any funds if:

1. Your business already exists in the UK and you are switching to an Innovator visa from another visa category
2. You have made innovations to the business and are already endorsed by the approved endorsing bodies

• Funds to support yourself: Fund of at least £1,270 is needed in your bank account for consecutive 28 days before you apply for an Innovator Visa.
• Knowledge of English: The applicant must usually prove their knowledge of English when they apply for UK Innovator Visa. The applicant’s must prove that they can write, speak, read and understand English to a level of B2 on the Common European Framework of Reference for Languages (CEFR) scale. This is considered an upper intermediate level which is equivalent to 5.5–6.5 on IELTS. They can prove this by the following:

1. Take a language test from an approved agency by passing SELT (Secure English Language Test)
2. Having a degree-level academic qualification that was taught in English at a University recognised in the UK
3. Having a GCSE, A Level, Scottish National Qualifications Level 4 or 5 or Scottish Higher or Advanced Higher in English
4. Having a degree level academic qualification which was taught in English

General Requirements for UK Innovator Visa

• At least 18 years old
• A valid business plan
• An endorsement letter from the endorsing bodies
• Being of good health
• Clean criminal records

UK Innovator Visa Benefits

• Path to Citizenship
• Permits Dual Citizenship
• Low/No Investment required
• Short processing time
• World-renowned schools and universities
• Enhanced Education opportunities
• Enhanced business access to large economies
• Visa-free travel to 180+ countries
• Live, work and study anywhere in the United Kingdom

Criteria for settlement of ILR

To get a permanent residence or Indefinite Leave to Remain (ILR) after the 3rd year the applicant must fulfil 2/7 requirements:

• Invested at least £50,000 in the business
• Generated minimum annual revenue of £1 Million dollars
• Generated minimum average revenue of £500,000 with at least £100,000 from overseas export
• Has applied for Intellectual Property Rights (IPR) in the United Kingdom
• Has created at least 10 full-time jobs for residents workers
• Has created at least five jobs for UK citizens with a gross salary of £25,000 a year
• You have at least doubled the numbers of customers in your business

For the purpose of getting a Permanent Residency the applicant must:

• You have to spend at least 180 days in 12 month period in the UK
• Demonstrate an active role in the day to day management of the business
• Submit a report to the endorsing body to showcase the development of the business after 6, 12 and 24 months of the business being operational.

UK Innovator Visa Application Fees:

Application Fee: The main applicant processing fee is £1,036 and

Include your Spouse/Children Fee: A processing fee of £1,036 per person

Application Timeline

• Month 1: Apply for a visa and collect it. Enter the UK and initiate the business venture
• Year 1–3: No more than 180 absent from the UK in each calendar year to be eligible for PR
• Year 5: PR granted. No more than 90 days absent from the county
• Year 6: UK Citizenship is granted provided no more than 450 days absent from the UK and more than 90 days in the last year

What you can Do and Cannot Do on the Innovator Visa

Can Do:

• Can set up several businesses
• Work for your business
• Bring your spouse and children as dependents
• Travel abroad and return back to the UK
• Can apply to settle permanently in the UK also known as ILR if you have been in the UK for 3 years
• Extend your visa multiple times

Cannot Do:

• You cannot work outside your business
• Work as a professional sportsperson or a sports coach
• Apply for public funds

Why choose the United Kingdom?

• Economic heart of the world
• Education: UK has one of the best educational institutions from pre-school to university
• Ease of travel and mobility: UK is one of the best countries for travel and mobility in this time of increased globalisation
• Healthcare: The UK is well known for its National Health Service, which provides excellent free health care. UK boasts some of the best medical facilities in the world
• Direct International flights connections to major global cities
• Attractive tax regime for High Net Worth Individuals

Written by Kaushik Sanghani (kaushik.sanghani@xumitcapital.com)

In this article, we will delve deeper into the concept of walk-forward analysis and its practical implementation. We will take the example of an exponential moving crossover system applied to NIFTY hourly data. Using Python, we will perform backtesting, optimization, and walk-forward analysis to evaluate the performance of the strategy. We will also discuss how to interpret the results and make decisions based on the findings. This will help readers understand the importance and usefulness of walk-forward analysis in evaluating trading strategies and models.

Before delving into the concept of walk-forward analysis, it’s important to understand why optimization is crucial in trading. Optimization is the process of finding the best combination of parameters for a trading strategy to ensure maximum profitability and minimum risk. For example, we may know that a combination of (20,50) exponential moving average crossover system works well in the live market and we are making consistent money by trading this strategy. But, there are thousands of other possible combinations we could have chosen, like (21,43) for example. To find the best combination that has worked out in historical data, we optimize our strategy.

However, using historical data to optimize a strategy has a pitfall called overfitting. It happens when a strategy is fine-tuned too much to the historical data, and it becomes too specific to that data. As a result, it may not work in the future, as we don’t have any evidence that this combination will work in the future or even if it works, we don’t have any test data to check this hypothesis.

To overcome this problem, we use some data for testing our trading strategy as out-of-sample data. We optimize the strategy based on in-sample data and find the suitable parameters for the strategy and check the validity of our strategy on out-of-sample data. This method works better than the previous one, but financial markets, particularly equity markets, are so dynamic that it may not be suitable to use 10–20 years back data for optimization purposes. Therefore, we need a dynamic technique to optimize our trading strategy, and this is where walk-forward analysis comes into play.

Walk-forward analysis:

Walk-forward analysis is a technique used to evaluate the performance of a trading strategy or model by simulating the process of repeatedly rolling forward the in-sample data and using the out-of-sample data to test the strategy. The goal is to avoid overfitting and to better assess the strategy’s ability to generalize to new data.

The process of walk-forward analysis begins by dividing the data into two parts: in-sample data and out-of-sample data. The in-sample data is used to train and optimize the strategy, while the out-of-sample data is used to test the strategy’s performance. The in-sample and out-of-sample data are typically divided using a rolling window, which moves forward in time as the analysis progresses.

Once the in-sample data has been used to train and optimize the strategy, the out-of-sample data is used to test the performance of the strategy. The process is then repeated by moving the rolling window forward in time and using the new out-of-sample data to test the strategy’s performance. This process is repeated multiple times, allowing the strategy to be tested on different periods of data.

The key advantage of walk-forward analysis is that it allows for a more realistic evaluation of the strategy’s performance. By simulating the process of repeatedly rolling forward the data and testing the strategy on new out-of-sample data, walk-forward analysis helps to avoid overfitting and to better assess the strategy’s ability to generalize to new data.

Backtesting in python

Now let’s understand walk-forward analysis with the use of python. We have taken hourly data from 2018–04–11 11:15:00 to 2022–11–29 15:30:00. Which are 8000 hourly data points. We will understand this analysis with the use of EMA crossover strategy. We will use vectorbt library for all the background calculations. See the chart to gain some context around the NIFTY 50 price.

We will begin by demonstrating how to use vectorbt, a powerful backtesting library, to perform backtesting on a random parameter of exponential moving averages (EMAs) with values (10, 20) and check the results. The code provided here can perform this task, and below it, the results of this strategy are compared against a buy and hold strategy as a benchmark. Vectorbt is particularly useful in this context because of its speed and its ability to provide a detailed report of trades and other parameters. It’s recommended to refer to the vectorbt documentation for more information on its capabilities and how to use it effectively.

#importing libraries
import pandas as pd  
import numpy as np 
import pickle
from datetime import datetime 
import vectorbt as vbt

#reading the datafile
df = pd.read_pickle(r'D://XumitCapital//Trading Strategy Backtesting Framework For Blog//nifty_1m_cleaned_data.pickle')

#Data pre-prcocessing
df = df.reset_index()
df['index'] = df['index'] - pd.Timedelta(minutes=1)
# Assuming you have a dataframe named df with a DatetimeIndex
df = df.set_index('index')
df = df.resample('60min', offset='15min').agg({'Open': 'first', 'High': 'max', 'Low': 'min', 'Close': 'last'})
df = df.dropna()
df = df.tail(8000)

#Here we are taking the recent last 8000 hours data 
df['Close'] = np.float64(df['Close'])
test_df = df.tail(1000) #1000 points for test purpose
train_df = df.head(7000) #7000 points for train purpose

import plotly.express as px

fig = px.line(df, x=df.index, y='Close', title='NIFTY hourly chart')
fig.show()

## Simple Just strategy making with random parameters

#calculating fast ema and slow ema
fast_ema = vbt.MA.run(close=train_df['Close'], window=10, ewm=True)
slow_ema = vbt.MA.run(close=train_df['Close'], window=20, ewm=True)

# Generate crossover signals
ema_entries = fast_ema.ma_crossed_above(slow_ema)
ema_exits = fast_ema.ma_crossed_below(slow_ema)

#creating portfolio
ema_pf = vbt.Portfolio.from_signals(train_df['Close'], ema_entries, ema_exits,freq='60m')

print(pd.DataFrame(ema_pf.stats(), columns=  ['Results']))

Now, we will optimize our strategy by adjusting the parameters of the fast EMA and slow EMA on the first 7000 data points. By doing this, we will find the best parameters and test them on 1000 out-of-sample data points. It is important to note that the process of selecting parameters and analyzing the results is a detailed topic that could require an entire article on its own. For now, let’s focus on the practical implementation of this process using Python. The code for this task can be found in the provided notebook, and it’s recommended to explore the code to understand the background calculations. Additionally, a heatmap of the results of different parameter combinations is provided below the code for further analysis.

## Optimization with min Window 10 and Max Window as 50

min_window = 10
max_window = 50

perf_metrics = ['total_return', 'positions.win_rate', 'positions.expectancy', 'max_drawdown']
perf_metric_names = ['Total return', 'Win rate', 'Expectancy', 'Max drawdown']

# Pre-calculate running windows on data with time buffer
fast_ma, slow_ma = vbt.MA.run_combs(
    train_df['Close'], np.arange(min_window, max_window+1), 
    r=2, short_names=['fast_ma', 'slow_ma'],ewm = True)

dmac_entries = fast_ma.ma_crossed_above(slow_ma)
dmac_exits = fast_ma.ma_crossed_below(slow_ma)

# Build portfolio
dmac_pf = vbt.Portfolio.from_signals(train_df['Close'], dmac_entries, dmac_exits,freq='60m')


# Calculate performance of each window combination
metric = 'total_return'

dmac_perf = dmac_pf.deep_getattr(metric)

print(dmac_perf.shape)

print(dmac_perf.idxmax())

print(dmac_perf.max())

##For plotting

# Convert this array into a matrix of shape (99, 99): 99 fast windows x 99 slow windows
dmac_perf_matrix = dmac_perf.vbt.unstack_to_df(symmetric=True, 
    index_levels='fast_ma_window', column_levels='slow_ma_window')


dmac_perf_matrix.vbt.heatmap(
    xaxis_title='Slow window', 
    yaxis_title='Fast window')

After the optimization process, we found that the combination of (10, 15) gave the best results on in-sample backtesting. Now, we will use this parameter combination to test it on the out-of-sample data points. The code and results for this are provided below. It’s important to note that results on out-of-sample data will give us a more accurate idea of how the strategy will perform in the future market conditions.

#calculating fast ema and slow ema
fast_ema = vbt.MA.run(close=test_df['Close'], window=10, ewm=True)
slow_ema = vbt.MA.run(close=test_df['Close'], window=15, ewm=True)

# Generate crossover signals
ema_entries = fast_ema.ma_crossed_above(slow_ema)
ema_exits = fast_ema.ma_crossed_below(slow_ema)

#creating portfolio
ema_pf = vbt.Portfolio.from_signals(test_df['Close'], ema_entries, ema_exits,freq='60m')

#printing the stats
print(ema_pf.stats())

Now, we will move on to the implementation of walk-forward analysis. In this process, we are using 8000 data points and creating 32 different windows, each containing 250 data points. From these windows, we take 200 as the in-sample data to run optimization on and test the best parameters on the out-of-sample data. It’s important to note that the best parameter combination is not only the one that gives the best returns but also the one that is most suitable for the market conditions. Below, you will find some visualizations and code to better understand this process.

# Creating the custom fixation for the  code
split_kwargs = dict(n=32,
                   window_len = 250,
                   set_lens = (50,),
                   left_to_right = False)

pf_kwargs = dict(direction = 'both',
                freq = '60m')

# Here n = 32 means we are dividing our total datapoints into 32 different windows, with each window lenght 250 from which
# 200 are insample data and 50 are out of the sample data


#we are considering windows from (10,50)
windows = np.arange(10,50)

#creating rolling split for each window
def roll_in_and_out_samples(price, **kwargs):
    return price.vbt.rolling_split(**kwargs)

#plotting each rolling split with in-sample and out-sample
roll_in_and_out_samples(df['Close'], **split_kwargs, plot=True, trace_names=['in-sample', 'out-sample'])

#saperating in-sample and out-sample and also indices in different dataframe
(in_price, in_indexes), (out_price, out_indexes) = roll_in_and_out_samples(df['Close'], **split_kwargs)

print(in_price.shape, len(in_indexes))  # in-sample
print(out_price.shape, len(out_indexes))  # out-sample


#creating the bench-mark or say holding from the start of the window till end of the in-sample datapoints
#0-200
in_hold_sharpe = simulate_holding(in_price, **pf_kwargs)
print(in_hold_sharpe)


#creating portfolio and also returning the total returns
#here we are considering the exponential moving averages
def simulate_all_params(price, windows, **kwargs):
    #to create dataframe for different combinations from our window (10,50)
    fast_ma, slow_ma = vbt.MA.run_combs(price, windows, r=2, short_names=['fast', 'slow'],ewm=True)
    #creating entries and exits while also creating portfolio and returning the total returns for comparision
    entries = fast_ma.ma_crossed_above(slow_ma)
    exits = fast_ma.ma_crossed_below(slow_ma)
    pf = vbt.Portfolio.from_signals(price, entries, exits, **kwargs)
    return pf.total_return()

# Simulate all params for in-sample ranges
in_sharpe = simulate_all_params(in_price, windows, **pf_kwargs)

print(in_sharpe)

#getting the best index from each period
def get_best_index(performance, higher_better=True):
    if higher_better:
        return performance[performance.groupby('split_idx').idxmax()].index
    return performance[performance.groupby('split_idx').idxmin()].index

in_best_index = get_best_index(in_sharpe)
print(in_best_index)

#getting the best params function for in-sample
def get_best_params(best_index, level_name):
    return best_index.get_level_values(level_name).to_numpy()


in_best_fast_windows = get_best_params(in_best_index, 'fast_window')
in_best_slow_windows = get_best_params(in_best_index, 'slow_window')
in_best_window_pairs = np.array(list(zip(in_best_fast_windows, in_best_slow_windows)))

print(in_best_window_pairs)


#plotting the best-params
pd.DataFrame(in_best_window_pairs, columns=['fast_window', 'slow_window']).vbt.plot()



#holding returns out of the sample data
out_hold_sharpe = simulate_holding(out_price, **pf_kwargs)
print(out_hold_sharpe)

# Simulate all params for out-sample ranges
out_sharpe = simulate_all_params(out_price, windows, **pf_kwargs)

print(out_sharpe)

#same we have done in in-sample
def simulate_best_params(price, best_fast_windows, best_slow_windows, **kwargs):
    fast_ma = vbt.MA.run(price, window=best_fast_windows, per_column=True)
    slow_ma = vbt.MA.run(price, window=best_slow_windows, per_column=True)
    entries = fast_ma.ma_crossed_above(slow_ma)
    exits = fast_ma.ma_crossed_below(slow_ma)
    pf = vbt.Portfolio.from_signals(price, entries, exits, **kwargs)
    return pf.total_return()


# Use best params from in-sample ranges and simulate them for out-sample ranges
out_test_sharpe = simulate_best_params(out_price, in_best_fast_windows, in_best_slow_windows, **pf_kwargs)

print(out_test_sharpe)
#Plotting 
cv_results_df = pd.DataFrame({
    'in_sample_hold': in_hold_sharpe.values,
    'in_sample_median': in_sharpe.groupby('split_idx').median().values,
    'in_sample_best': in_sharpe[in_best_index].values,
    'out_sample_hold': out_hold_sharpe.values,
    'out_sample_median': out_sharpe.groupby('split_idx').median().values,
    'out_sample_test': out_test_sharpe.values
})


color_schema = vbt.settings['plotting']['color_schema']

cv_results_df.vbt.plot(
    trace_kwargs=[
        dict(line_color=color_schema['blue']),
        dict(line_color=color_schema['blue'], line_dash='dash'),
        dict(line_color=color_schema['blue'], line_dash='dot'),
        dict(line_color=color_schema['orange']),
        dict(line_color=color_schema['orange'], line_dash='dash'),
        dict(line_color=color_schema['orange'], line_dash='dot')
    ]
)

Different windows and their in-sample and out-sample visulization

Optimization results on on each window from in-sample data

Results from out-sample backtesting on each window:

Visualization of mean and median parameters:

Visit our website at: xumitcapital.com

A Message from InsiderFinance

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the InsiderFinance Wire
• 📚 Take our FREE Masterclass
• 📈 Discover Powerful Trading Tools

Advanced backtesting on NIFTY using Walk-forward Analysis in Python was originally published in InsiderFinance Wire on Medium, where people are continuing the conversation by highlighting and responding to this story.