You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.
Search engines attempt to rank results for a given search based on their relevance to the topic, and the quality and reliability a site is judged to have. Google, the world’s most popular search engine, uses an ever-evolving algorithm that aims to evaluate sites in the way that a human reader would. This means that a key part of SEO involves ensuring that the website is a unique and relevant resource for readers.
Targeting, viewability, brand safety and invalid traffic: Targeting, viewability, brand safety and invalid traffic all are aspects used by marketers to help advocate digital advertising. Cookies are a form of digital advertising, which are tracking tools within desktop devices; causing difficulty, with shortcomings including deletion by web browsers, the inability to sort between multiple users of a device, inaccurate estimates for unique visitors, overstating reach, understanding frequency, problems with ad servers, which cannot distinguish between when cookies have been deleted and when consumers have not previously been exposed to an ad. Due to the inaccuracies influenced by cookies, demographics in the target market are low and vary (Whiteside, 2016). Another element, which is affected within digital marketing, is ‘viewabilty’ or whether the ad was actually seen by the consumer. Many ads are not seen by a consumer and may never reach the right demographic segment. Brand safety is another issue of whether or not the ad was produced in the context of being unethical or having offensive content. Recognizing fraud when an ad is exposed is another challenge marketers face. This relates to invalid traffic as premium sites are more effective at detecting fraudulent traffic, although non-premium sites are more so the problem (Whiteside, 2016).
Understanding Mobiles: Understanding mobile devices is a significant aspect of digital marketing because smartphones and tablets are now responsible for 64% of the time US consumers are online (Whiteside, 2016). Apps provide a big opportunity as well as challenge for the marketers because firstly the app needs to be downloaded and secondly the person needs to actually use it. This may be difficult as ‘half the time spent on smartphone apps occurs on the individuals single most used app, and almost 85% of their time on the top four rated apps’ (Whiteside, 2016). Mobile advertising can assist in achieving a variety of commercial objectives and it is effective due to taking over the entire screen, and voice or status is likely to be considered highly; although the message must not be seen or thought of as intrusive (Whiteside, 2016). Disadvantages of digital media used on mobile devices also include limited creative capabilities, and reach. Although there are many positive aspects including the users entitlement to select product information, digital media creating a flexible message platform and there is potential for direct selling (Belch & Belch, 2012).
Once consumers can access this content, they want to engage with something that fits their needs and is sensory and interactive — from the early popularity of web portals to the spread of online video, to the next generation virtual realities. Their digital desires are marked by a thirst for content. The old media adage that “content is king" is correct. There is no question that the desire to engage with content is a key driver of customer behavior.
Another ethical controversy associated with search marketing has been the issue of trademark infringement. The debate as to whether third parties should have the right to bid on their competitors' brand names has been underway for years. In 2009 Google changed their policy, which formerly prohibited these tactics, allowing 3rd parties to bid on branded terms as long as their landing page in fact provides information on the trademarked term. Though the policy has been changed this continues to be a source of heated debate.
Robots.txt is not an appropriate or effective way of blocking sensitive or confidential material. It only instructs well-behaved crawlers that the pages are not for them, but it does not prevent your server from delivering those pages to a browser that requests them. One reason is that search engines could still reference the URLs you block (showing just the URL, no title or snippet) if there happen to be links to those URLs somewhere on the Internet (like referrer logs). Also, non-compliant or rogue search engines that don't acknowledge the Robots Exclusion Standard could disobey the instructions of your robots.txt. Finally, a curious user could examine the directories or subdirectories in your robots.txt file and guess the URL of the content that you don't want seen.
At HubSpot, we talk a lot about inbound marketing as a really effective way to attract, engage, and delight customers online. But we still get a lot of questions from people all around the world about digital marketing. So, we decided to answer them. Click the links below to jump to each question, or keep reading to see how digital marketing is carries out today.