Web crawler questions

Panscient operates a large-scale web crawler which crawls millions of websites on a regular basis. Similar to the web crawlers used by the large search engines, our web crawler crawls public websites looking for specific types of information to include in vertical search engines.

What kind of information are you crawling for?

Panscient primarily crawls the web looking for corporate information, such as company names, addresses, executive biographies, job openings and product information. We also crawl the web to locate genealogy pages, such as birth, marriage and death records, obituaries and census records.

Are you crawling for private information?

Our web crawler only accesses publicly available information published on websites. We respect the rights of website owners to control what content our crawler analyzes. Our crawler obeys the Robot Exclusion Standard, and will not collect content from any pages that are off-limits to robots.

How did your crawler find my website?

We crawl the entire list of registered .com domain names and several smaller top-level domains, which are publicly available through Verisign. Once you register a domain name, our crawler will periodically check it for business information.

How can I control the information your crawler collects from my website?

The Panscient web crawler identifies itself using the user-agents "panscient.com" or "pantest", and obeys the Robot Exclusion Standard. To exclude the Panscient web crawler from accessing portions of your site, please modify your website's robots.txt file to identify the directories and files which the crawler should not request. Our web crawler also obeys the robots meta-tag directives of "noindex" and "nofollow", which can be placed in the header section of individual web pages.

To completely exclude our web crawler from your site, add the following entry to your robots.txt file:

User-Agent: panscient.com
Disallow: /

Why is your web crawler trying to access pages that don't exist on my website?

Our web crawler attempts to extract links to valid web pages from javascript and other scripting languages. The crawler may misinterpret the information in these scripts and request a page that does not actually exist. These requests are attempts to retrieve valid web content, and are not an attempt to circumvent your webserver security.

How often will your crawler request a page from my server?

The Panscient web crawler will request a page at most once every second from the same domain name or the same IP address.

I still have a question or concern about your web crawler. What should I do?

Contact us at crawler@panscient.com and we will respond to your query promptly.

California Residents

California Consumer Privacy Act Notice

1. About this Privacy Notice

   Pursuant to the California Consumer Privacy Act of 2018 (“CCPA”), Panscient Inc. (“Panscient,” “we,” “us” or “our”) are providing this California Consumer Privacy Act Notice (“CCPA Notice”). This CCPA Notice supplements our Privacy Policy and applies solely to California consumers, effective January 1, 2023. Terms used but not defined shall have the meaning ascribed to them in the CCPA.

2. Personal Information Collected, Disclosed, or Sold

   We collect the following categories of personal information, as indicated below:

   Category	Examples	Collected, sold and disclosed
   A. Identifiers.	A real name, alias, postal address, unique personal identifier, online identifier, Internet Protocol address, email address, account name, Social Security number, driver's license number, passport number, or other similar identifiers.	We may have collected some of these data elements on California Residents.
   B. Personal information categories listed in the California Customer Records statute (Cal. Civ. Code § 1798.80(e)).	A name, signature, Social Security number, physical characteristics or description, address, telephone number, passport number, driver's license or state identification card number, insurance policy number, education, employment, employment history, bank account number, credit card number, debit card number, or any other financial information, medical information, or health insurance information. Some personal information included in this category may overlap with other categories.	We may have collected some of these data elements on California Residents.
   C. Protected classification characteristics under California or federal law.	Age (40 years or older), race, color, ancestry, national origin, citizenship, religion or creed, marital status, medical condition, physical or mental disability, sex (including gender, gender identity, gender expression, pregnancy or childbirth and related medical conditions), sexual orientation, veteran or military status, genetic information (including familial genetic information).	We do not collect this type of information.
   D. Commercial information.	Records of personal property, products or services purchased, obtained, or considered, or other purchasing or consuming histories or tendencies.	We do not collect this type of information.
   E. Biometric information.	Genetic, physiological, behavioral, and biological characteristics, or activity patterns used to extract a template or other identifier or identifying information, such as, fingerprints, faceprints, and voiceprints, iris or retina scans, keystroke, gait, or other physical patterns, and sleep, health, or exercise data.	We do not collect this type of information.
   F. Internet or other similar network activity.	Browsing history, search history, information on a consumer's interaction with a website, application, or advertisement.	We do not collect this type of information.
   G. Geolocation data.	Physical location or movements.	We do not collect this type of information.
   H. Sensory data.	Audio, electronic, visual, thermal, olfactory, or similar information.	We do not collect this type of information.
   I. Professional or employment-related information.	Current or past job history or performance evaluations.	We may have collected some of these data elements on California Residents.
   J. Non-public education information (per the Family Educational Rights and Privacy Act (20 U.S.C. Section 1232g, 34 C.F.R. Part 99)).	Education records directly related to a student maintained by an educational institution or party acting on its behalf, such as grades, transcripts, class lists, student schedules, student identification codes, student financial information, or student disciplinary records.	We do not collect this type of information.
   K. Inferences drawn from other personal information.	Profile reflecting a person's preferences, characteristics, psychological trends, predispositions, behavior, attitudes, intelligence, abilities, and aptitudes.	We do not collect this type of information.

   We collect this personal information from publicly available web sources.

   We share this personal information with: our affiliates; our customers; our vendors, service providers, suppliers, agents and representatives; joint venture partners; resellers, distributors, referral partners and integrators; and other parties where required by law or to protect our rights.

   We may also sell this personal information to our affiliates; our customers; our vendors; joint venture partners; resellers, distributors, referral partners and integrators.

   We may use this personal information to operate, manage, and maintain our business, to provide our products and services, to communicate with you, for our vendor management purposes, and to accomplish our business purposes and objectives, including, for example, using personal information to: develop, improve, repair, and maintain our products and services; process or fulfill a request or other transactions submitted to us; personalize, advertise, and market our products and services; conduct research, analytics, and data analysis; maintain our facilities and infrastructure; undertake quality and safety assurance measures; conduct risk and security control and monitoring; detect and prevent fraud; perform identity verification; perform accounting, audit, and other internal functions; comply with law, legal process, and internal policies; maintain records; exercise and defend legal claims; and fulfill legal obligations.

3. Consumer Rights

   You have the right to request that we disclose to you
   1. the categories of personal information we collected about you and the categories of sources from which we collected such information;
   2. the specific pieces of personal information we collected about you;
   3. the business or commercial purpose for collecting personal information about you; and
   4. the categories of personal information about you that we shared or disclosed and the categories of third parties with whom we shared or to whom we disclosed such information in the preceding 12 months.

   You also have the right to request that we delete personal information we collected from you subject to certain exceptions.

   You also have the right to not be discriminated against in pricing and services because you exercise any of your rights under the CCPA. Panscient does not offer financial incentives or price or service differences to consumers in exchange for the retention or sale of a consumer’s personal information.

4. How to Make a Request

   You may make a request for the disclosures or deletion described above by emailing ccpa@panscient.com or via this form.

   You may be required to submit proof of your identity for these requests to be processed as a verifiable consumer request. We may not be able to comply with your request if we are unable to confirm your identity or to connect the information you submit in your request with personal information in our possession. You may designate an authorized agent to make a request on your behalf subject to proof of identity and authorization.

   We will respond to your request consistent with the CCPA, which does not apply to certain information, such as information made available from government records, certain data subject to the Fair Credit Reporting Act (FCRA), Gramm-Leach-Bliley Act (GLBA) and certain other laws, and where its application is preempted by, or in conflict with, federal law or the United States or California Constitution.

5. Changes

   We will update this CCPA Notice from time to time. Any changes will be posted on this page with an updated revision date.

Last updated: June 3, 2023