WeDataSphere Getting Started Guide
Since the open sourcing of WeDataSphere in 2019, the project has experienced rapid growth. The total number of users in the WeDataSphere and Apache Linkis open source community groups has exceeded 7,600, with over 2,600 companies using the sandbox, and feedback from more than 110 companies that have deployed it in production. The production environment supports over 400PB of data and serves over 5,000 users, spanning various industries such as finance, telecommunications, manufacturing, and the internet.
In order to help newcomers better understand WeDataSphere, we have written this article to help them quickly get started with WeDataSphere.
1. WeDataSphere Introduction
WeDataSphere is a financial level one-stop open-source suitcase for big data platforms. The fundamental platform consists of 4 layers for data exchange, data distribution, computation and storage; The functional platform consists of 3 layers for platform tools, data tools and application tools, focusing on the implementations of various user requirements about functional tools. These construct as a complete technical ecosystem of big data platform and provides one-stop sufficient components and functionalities support.
After years of development, WeDataSphere has formed its own core advantages:
● One-stop: Offers a wide range of functional components from data application development to data visualization, from batch jobs to real-time streaming computing capabilities to meet different scenarios of data application development, operation, and data management requirements.
● Financial-grade: Enhancements have been made in the areas of high availability, data governance, and data security, to create a highly reliable financial-grade big data platform to support core critical business applications.
● Fully-connected: The unique data application development management portal DataSphere Studio and computational middleware Linkis, with the two-layer integrated architecture, enable true communication between the components of the platform, providing a more seamless user experience, simplified platform architecture, and stronger control functions.
● Open Source: WeDataSphere is based on open source and returns to open source. The various components developed in-house will be gradually open sourced. The overall design is open and flexible, with friendly extensibility and pluggable components. It also attracts more individuals and organizations to participate in the development, construction, and promotion of WDS through an open-source approach.
2. GitHub
Currently, all of our open-source projects can be found on GitHub. The following are the project addresses for quick queries:
● WeDataSphere Big Data Platform Suite: https://github.com/WeBankFinTech/WeDataSphere/
● Apache Linkis Computing Middleware: https://github.com/apache/incubator-linkis
● DataSphereStudio One-Stop Data Application Development Management Portal: https://github.com/WeBankFinTech/DataSphereStudio
● Qualitis Data Quality Management Platform: https://github.com/WeBankFinTech/Qualitis
● Schedulis Workflow Task Scheduling System: https://github.com/WeBankFinTech/Schedulis
● Exchangis Data Exchange Platform: https://github.com/WeBankFinTech/Exchangis
● Prophecis One-Stop Machine Learning Platform: https://github.com/WeBankFinTech/Prophecis
● Scriptis Interactive Data Analysis Web Tool: https://github.com/WeBankFinTech/Scriptis
● Visualis Data Visualization Tool: https://github.com/WeBankFinTech/Visualis
● Streamis Streaming Application Development Management System: https://github.com/WeBankFinTech/Streamis
3. QA Documentation
Many developers new to WeDataSphere may encounter various issues during the installation and deployment process. Therefore, it is recommended to follow the methods below for inquiries:
(1)Check the QA documentation to see if there are similar problems and solutions. QA document link: https://docs.qq.com/doc/DSGZhdnpMV3lTUUxq?u=e664e79acf4f467aa99c565ea317e288
(2)Seek community answers through the following four ways:
Get WeBank CoreTech’s stories in your inbox
Join Medium for free to get updates from this writer.
● Ask a question through Issues: Raise an Issue in the corresponding project’s GitHub address and provide a detailed description of the problem encountered;
● Ask a question via email: Subscribe to the Apache mailing list and ask questions by sending an email. Subscription strategy link: https://linkis.apache.org/zh-CN/community/how-to-subscribe
● Ask a question in the QA document: Directly enter the problem encountered in the QA document
● Ask a question in the community: Raise questions in QQ and WeChat groups
After you initiate a question, please be patient. Community developers will provide answers through the above four channels.
4. Sandbox Environment Experience
To help new users better understand WeDataSphere’s functionality, the community provides a sandbox environment for experiencing the platform. After registering, users can log in and use it.
Sandbox environment website: https://dss-open.wedatasphere.com/
Login account/password: Registered phone number/last six digits of the phone number
5. Learning Materials
● Technical articles: Covering various fields such as installation and deployment, practical use, and development guides. Click on the following article to go to the article collection: “WeDataSphere Community Article Collection”
● Video tutorials: https://space.bilibili.com/598542776
6. How to Participate in the Community
All activity plans will be continuously updated, including but not limited to the following participation methods. If you are interested in community activities, please follow or contact the WeDataSphere assistant.
(1) Participate in the biweekly community meetings: Every two weeks, the community will regularly hold developer meetings, which are open to anyone interested in the community. For participation guidelines, please refer to the article: “Participate in Apache Linkis Community Developer Biweekly Meetings and Grow with the Community”
(2) Community solicitations: Contributions can include but are not limited to production cases, source code analysis, application examples, and tutorial demonstrations.
(3) Community activities: Regularly post community activities in the community groups and encourage active participation from community users.
(4) Become a guest speaker: Outstanding community contributors have the opportunity to be invited to participate in community sharing. At the same time, you can actively contact the assistant to become a guest speaker based on project experience.
(5) Become a community evangelist: Outstanding community members have the opportunity to be invited to become community evangelists, or actively contact the assistant to become a community evangelist.
7. How to Contribute to the Community
Main Contribution Method Description
● Beginner tasks: Claim beginner tasks, see details at https://github.com/apache/incubator-linkis/issues/1161
● Documentation contributions: The completion and improvement of documentation are crucial to the development of the community. We hope you can find areas that need optimization and improvement in the existing documentation. The content includes but is not limited to installation and deployment, usage experience, and case practices. You can submit your articles to the assistant or enter them in the corresponding project issues, such as Linkis https://github.com/apache/incubator-linkis/issues/1233
● Code contributions: Includes but is not limited to bug feedback and fixes, and submitting PRs.
● Community Q&A: Helping answer questions is a very valuable way to contribute to the community. There will be new members and partners joining the community, and helping them is an excellent opportunity to showcase your expertise.
● Become a community volunteer: Non-code contributions include but are not limited to website translation, editing and layout, submissions, speaking, and organizing activities.
About Us
We are WeBank CoreTech, the core technology team of WeBank. Our team has incubated top-tier open-source projects such as Apache Linkis and Apache EventMesh. For any inquiries or collaborations, please contact us.
Email: webankcoretech@gmail.com
Twitter: @WebankCoreTech.
